[jira] [Commented] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually

[Kostya Golikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804004#comment-13804004
 ] 

Kostya Golikov commented on KAFKA-1079:
---

1. It is common convention in Scala to distinguish pure no arg functions from 
side effecting ones. Since current implementation of choosePort creates 
socket(s) and moreover can yield different results on different runs, I guess 
it is pretty reasonable to put braces next to the method call. 
2. Okay, I will revert this change. 

> Liars in PrimitiveApiTest that promise to test api in compression mode, but 
> don't do this actually
> --
>
> Key: KAFKA-1079
> URL: https://issues.apache.org/jira/browse/KAFKA-1079
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Affects Versions: 0.8
>Reporter: Kostya Golikov
>Priority: Minor
>  Labels: newbie, test
> Attachments: testing-with-compression-producer.patch
>
>
> Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's 
> allowed us to control compression. Times goes on and now PrimitiveApiTest 
> have methods that promise to test api with compression enabled, but in fact 
> they don't. Moreover this methods almost entirely copy their counterparts 
> without compression. In particular I'm talking about 
> `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and 
> `testMultiProduce`/`testMultiProduceWithCompression` pairs. 
> The fix could be super-easy and soundness -- just parameterize methods with 
> producer of each type (with/without compression). Sadly but it isn't feasible 
> for junit3, so straightforward solution is to do the same ugly thing as 
> `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget 
> about class-wide producer and roll-out it's own. I will attach path if that 
> is a problem indeed. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 14865: Patch for KAFKA-1097

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14865/#review27459
---


Thanks for the patch. There a couple of issues.

1. The main one is during the phase of partition reassignment when we bootstrap 
new replicas. At this point, the assigned replica list doesn't include the new 
replicas. If we only allow replicas in assigned replica set to be added to ISR, 
those new replicas won't be added to ISR, which will prevent partition 
reassignment from completing. We could include those new replicas in the all 
replica set in the LeaderAndIsr request. We probably have to think a bit more 
to see if there is any other impact.

2. Once the assigned replica set in the broker is updated. We need to prevent 
an old replica from being added back to this set again. Currently, in 
Partition.updateLeaderHWAndMaybeExpandIsr() (triggered by a fetch request), it 
will call getOrCreateReplica(), which can cause a replica to be added back to 
the assigned replica set. What we can do is to only call getOrCreateReplica 
during makeLeader() and makeFollower(). In the former, we force all replicas to 
be created. In the latter, we just need to make sure the local replica is 
created. In Partition.updateLeaderHWAndMaybeExpandIsr(), we can then use 
getReplica(), instead of getOrCreateReplica().

- Jun Rao


On Oct. 23, 2013, 4:34 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14865/
> ---
> 
> (Updated Oct. 23, 2013, 4:34 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1097
> https://issues.apache.org/jira/browse/KAFKA-1097
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1097 Race condition while reassigning low throughput partition leads to 
> incorrect ISR information in zookeeper; The changes include 1) Adding the ISR 
> shrink logic as part of the OfflineReplica -> NonExistentReplica state change 
> 2) Adding a safety check on the broker where it only expands the ISR if the 
> replica is in the assigned replica list 3) Updating the assigned replica list 
> on the broker on every makeLeader request and also on makeFollower request 
> for safety, though that's not strictly required. These changes will ensure 
> that the ISR is shrunk by the controller and the leader has an updated 
> assigned replica list. So even if a replica sends a fetch request after the 
> ISR is shrunk by the controller, the broker will not be able to update the 
> ISR until it receives the next LeaderAndIsrRequest (which notifies it of the 
> latest zkVersion of the partition state path) that also contains the shrunk 
> ISR and assigned replica list. Using that the broker will avoid expanding the 
> ISR i
 f the replica is not present in the new assigned replica list
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 60f3ed4e88b08d0dcf2ea84259ae6dbe9d7c3a2d 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 88d130f55997b72a8590e4cfe92857a7320e70d5 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 212c05d65dcdc147e55f90875bacc940e30342bf 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 03ba60e82cdb3dce100603d615894ede47e4b077 
>   kafka-patch-review.py 82ea9a890fe79aad7d0ea6d33f3e2780e036317c 
> 
> Diff: https://reviews.apache.org/r/14865/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 14898: Patch for KAFKA-1060

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14898/#review27460
---



core/src/main/scala/kafka/network/SocketServer.scala


Wouldn't it be better if this is done inside 
requestChannel.receiveResponse()?



- Jun Rao


On Oct. 24, 2013, 1:43 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14898/
> ---
> 
> (Updated Oct. 24, 2013, 1:43 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1060
> https://issues.apache.org/jira/browse/KAFKA-1060
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1060.v1
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/network/RequestChannel.scala 
> 77d7ec034d8534c0d3aef49e9516b9b853c99cb2 
>   core/src/main/scala/kafka/network/SocketServer.scala 
> 216245deb119c8f5e8710b25d5093c456398c568 
> 
> Diff: https://reviews.apache.org/r/14898/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Commented] (KAFKA-1101) Need better java api for embedding kafkaserver in a java container app

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804374#comment-13804374
 ] 

Jun Rao commented on KAFKA-1101:


That's fair. Probably we should patch KafkaServerStartable so that it 
propagates the exception back to the caller. The caller can decide what to do 
with it.

> Need better java api for embedding kafkaserver in a java container app
> --
>
> Key: KAFKA-1101
> URL: https://issues.apache.org/jira/browse/KAFKA-1101
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Rosenberg
>
> We embed the KafkaServer inside a java service container framework, which 
> makes it easy to deploy and monitor within our infrastructure.  When 
> upgrading to kafka 0.8 from 0.7.2, I ran into an issue with not being able to 
> pass the needed constructor arg (SystemTime), since there doesn't appear to 
> be an easy way to instantiate that from java.  So, I ended up with this janky 
> solution using SystemTime$.MODULE$.
> Could a default constructor be added which assumes a default SystemTime, 
> rather than requiring that here?
> Note, I need to construct a KafkaServer directly, since I need to manage the 
> lifecycle more directly than can be done with KafkaServerStartable.
> {code}
>   // Need to do janky scala MODULE$ dereferencing, in order to get a 
> default value in ctor
>   server = new kafka.server.KafkaServer(kafkaConfig, SystemTime$.MODULE$);
>   server.startup();
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1097) Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804367#comment-13804367
 ] 

Jun Rao commented on KAFKA-1097:


Posted the comment on the RB. The patch is a bit tricky. I agree that we should 
patch it only in trunk so that we will can test it out more thoroughly.

> Race condition while reassigning low throughput partition leads to incorrect 
> ISR information in zookeeper 
> --
>
> Key: KAFKA-1097
> URL: https://issues.apache.org/jira/browse/KAFKA-1097
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.8
>
> Attachments: KAFKA-1097.patch
>
>
> While moving partitions, the controller moves the old replicas through the 
> following state changes -
> ONLINE -> OFFLINE -> NON_EXISTENT
> During the offline state change, the controller removes the old replica and 
> writes the updated ISR to zookeeper and notifies the leader. Note that it 
> doesn't notify the old replicas to stop fetching from the leader (to be fixed 
> in KAFKA-1032). During the non-existent state change, the controller does not 
> write the updated ISR or replica list to zookeeper. Right after the 
> non-existent state change, the controller writes the new replica list to 
> zookeeper, but does not update the ISR. So an old replica can send a fetch 
> request after the offline state change, essentially letting the leader add it 
> back to the ISR. The problem is that if there is no new data coming in for 
> the partition and the old replica is fully caught up, the leader cannot 
> remove it from the ISR. That lets a non existent replica live in the ISR at 
> least until new data comes in to the partition



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1079) Liars in PrimitiveApiTest that promise to test api in compression mode, but don't do this actually

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804378#comment-13804378
 ] 

Jun Rao commented on KAFKA-1079:


For 1, the convention we have been following is that we will only omit the 
braces if the function has no side effect. In this particular case, 
choosePort() does have side effect. So, we should include the braces.

> Liars in PrimitiveApiTest that promise to test api in compression mode, but 
> don't do this actually
> --
>
> Key: KAFKA-1079
> URL: https://issues.apache.org/jira/browse/KAFKA-1079
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Affects Versions: 0.8
>Reporter: Kostya Golikov
>Priority: Minor
>  Labels: newbie, test
> Attachments: testing-with-compression-producer.patch
>
>
> Long time ago (0.7) we had ByteBufferMessageSet as a part of api and it's 
> allowed us to control compression. Times goes on and now PrimitiveApiTest 
> have methods that promise to test api with compression enabled, but in fact 
> they don't. Moreover this methods almost entirely copy their counterparts 
> without compression. In particular I'm talking about 
> `testProduceAndMultiFetch` / `testProduceAndMultiFetchWithCompression` and 
> `testMultiProduce`/`testMultiProduceWithCompression` pairs. 
> The fix could be super-easy and soundness -- just parameterize methods with 
> producer of each type (with/without compression). Sadly but it isn't feasible 
> for junit3, so straightforward solution is to do the same ugly thing as 
> `testDefaultEncoderProducerAndFetchWithCompression` method does -- forget 
> about class-wide producer and roll-out it's own. I will attach path if that 
> is a problem indeed. 



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1097) Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1097:
-

Fix Version/s: (was: 0.8)
   0.8.1

> Race condition while reassigning low throughput partition leads to incorrect 
> ISR information in zookeeper 
> --
>
> Key: KAFKA-1097
> URL: https://issues.apache.org/jira/browse/KAFKA-1097
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.8.1
>
> Attachments: KAFKA-1097.patch
>
>
> While moving partitions, the controller moves the old replicas through the 
> following state changes -
> ONLINE -> OFFLINE -> NON_EXISTENT
> During the offline state change, the controller removes the old replica and 
> writes the updated ISR to zookeeper and notifies the leader. Note that it 
> doesn't notify the old replicas to stop fetching from the leader (to be fixed 
> in KAFKA-1032). During the non-existent state change, the controller does not 
> write the updated ISR or replica list to zookeeper. Right after the 
> non-existent state change, the controller writes the new replica list to 
> zookeeper, but does not update the ISR. So an old replica can send a fetch 
> request after the offline state change, essentially letting the leader add it 
> back to the ISR. The problem is that if there is no new data coming in for 
> the partition and the old replica is fully caught up, the leader cannot 
> remove it from the ISR. That lets a non existent replica live in the ISR at 
> least until new data comes in to the partition



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-918) Change log.retention.hours to be log.retention.mins

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804389#comment-13804389
 ] 

Jun Rao commented on KAFKA-918:
---

Thanks for the patch. Looks good. Could you remove unused imports in 
KafkaConfigTest?

> Change log.retention.hours to be log.retention.mins
> ---
>
> Key: KAFKA-918
> URL: https://issues.apache.org/jira/browse/KAFKA-918
> Project: Kafka
>  Issue Type: New Feature
>  Components: config
>Affects Versions: 0.7.2
>Reporter: Jason Weiss
>  Labels: features, newbie
> Attachments: issue_918.patch
>
>
> We stood up a cluster that is processing over 350,000 events per second, with 
> each event a fixed payload size of 2K. The storage required to process that 
> much data over an hour is beyond what we wanted to pay for at AWS. 
> Additionally, we don't have a requirement to keep the files around for an 
> extended period after processing.
> It would be tremendously valuable for us to be able to define the 
> log.retention in minutes, not hours. For example, we would prefer to only 
> keep 30 minutes of logs around.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1100) metrics shouldn't have generation/timestamp specific names

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804411#comment-13804411
 ] 

Jun Rao commented on KAFKA-1100:


Yes, I think this is too late for 0.8 final.

> metrics shouldn't have generation/timestamp specific names
> --
>
> Key: KAFKA-1100
> URL: https://issues.apache.org/jira/browse/KAFKA-1100
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Rosenberg
>
> I've noticed that there are several metrics that seem useful for monitoring 
> overtime, but which contain generational timestamps in the metric name.
> We are using yammer metrics libraries to send metrics data in a background 
> thread every 10 seconds (to kafka actually), and then they eventually end up 
> in a metrics database (graphite, opentsdb).  The metrics then get graphed via 
> UI, and we can see metrics going way back, etc.
> Unfortunately, many of the metrics coming from kafka seem to have metric 
> names that change any time the server or consumer is restarted, which makes 
> it hard to easily create graphs over long periods of time (spanning app 
> restarts).
> For example:
> names like: 
> kafka.consumer.FetchRequestAndResponseMetricssquare-1371718712833-e9bb4d10-0-508818741-AllBrokersFetchRequestRateAndTimeMs
> or: 
> kafka.consumer.ZookeeperConsumerConnector...topicName.square-1373476779391-78aa2e83-0-FetchQueueSize
> In our staging environment, we have our servers on regular auto-deploy cycles 
> (they restart every few hours).  So just not longitudinally usable to have 
> metric names constantly changing like this.
> Is there something that can easily be done?  Is it really necessary to have 
> so much cryptic info in the metric name?



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: preparing for the 0.8 final release

[Jun Rao]

At this moment, we have resolved all jiras that we intend to fix in 0.8.0
final.

Joe,

Would you like to drive the 0.8.0 final release again?

Thanks,

Jun


On Mon, Oct 21, 2013 at 8:53 PM, Jun Rao  wrote:

> Hi, Everyone,
>
> At this moment, we have only one remaining jira (KAFKA-1097) that we plan
> to fix in 0.8. After that, we can cut the final 0.8 release.
>
> Thanks,
>
> Jun
>
>
>
> On Mon, Oct 7, 2013 at 5:33 PM, Jun Rao  wrote:
>
>> Hi, Everyone,
>>
>> I made another pass of the remaining jiras that we plan to fix in the 0.8
>> final release.
>>
>>
>> https://issues.apache.org/jira/browse/KAFKA-954?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%20%220.8%22%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)
>>
>> Do people agree with this list?
>>
>> Joe,
>>
>> I don't have good understanding of KAFKA-1018. Do you think this needs to
>> be fixed in 0.8 final?
>>
>> Thanks,
>>
>> Jun
>>
>>
>>
>> On Fri, Sep 13, 2013 at 9:18 AM, Jun Rao  wrote:
>>
>>> Hi, Everyone,
>>>
>>> We have been stabilizing the 0.8 branch since the beta1 release. I think
>>> we are getting close to an 0.8 final release. I made an initial list of the
>>> remaining jiras that should be fixed in 0.8.
>>>
>>>
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%20%220.8%22%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)
>>>
>>> 1. Do people agree with the list?
>>>
>>> 2. If the list is good, could people help contributing/reviewing the
>>> remaining jiras?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>
>>
>

[jira] [Commented] (KAFKA-1101) Need better java api for embedding kafkaserver in a java container app

[Jason Rosenberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804425#comment-13804425
 ] 

Jason Rosenberg commented on KAFKA-1101:


The other issue with KafkaServerStartable is that it calls shutdown() if the 
startup fails, but then there can be non-daemon threads left running which 
prevent a clean shutdown.  I think I found this, and filed KAFKA-589 a while 
back (which was one of the main reason I moved away from using 
KafkaServerStartable).

> Need better java api for embedding kafkaserver in a java container app
> --
>
> Key: KAFKA-1101
> URL: https://issues.apache.org/jira/browse/KAFKA-1101
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Rosenberg
>
> We embed the KafkaServer inside a java service container framework, which 
> makes it easy to deploy and monitor within our infrastructure.  When 
> upgrading to kafka 0.8 from 0.7.2, I ran into an issue with not being able to 
> pass the needed constructor arg (SystemTime), since there doesn't appear to 
> be an easy way to instantiate that from java.  So, I ended up with this janky 
> solution using SystemTime$.MODULE$.
> Could a default constructor be added which assumes a default SystemTime, 
> rather than requiring that here?
> Note, I need to construct a KafkaServer directly, since I need to manage the 
> lifecycle more directly than can be done with KafkaServerStartable.
> {code}
>   // Need to do janky scala MODULE$ dereferencing, in order to get a 
> default value in ctor
>   server = new kafka.server.KafkaServer(kafkaConfig, SystemTime$.MODULE$);
>   server.startup();
> {code}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: preparing for the 0.8 final release

[Joe Stein]

Sounds good, yup!


/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop
/


On Oct 24, 2013, at 1:12 PM, Jun Rao  wrote:

> At this moment, we have resolved all jiras that we intend to fix in 0.8.0
> final.
> 
> Joe,
> 
> Would you like to drive the 0.8.0 final release again?
> 
> Thanks,
> 
> Jun
> 
> 
> On Mon, Oct 21, 2013 at 8:53 PM, Jun Rao  wrote:
> 
>> Hi, Everyone,
>> 
>> At this moment, we have only one remaining jira (KAFKA-1097) that we plan
>> to fix in 0.8. After that, we can cut the final 0.8 release.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> 
>> 
>> On Mon, Oct 7, 2013 at 5:33 PM, Jun Rao  wrote:
>> 
>>> Hi, Everyone,
>>> 
>>> I made another pass of the remaining jiras that we plan to fix in the 0.8
>>> final release.
>>> 
>>> 
>>> https://issues.apache.org/jira/browse/KAFKA-954?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%20%220.8%22%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)
>>> 
>>> Do people agree with this list?
>>> 
>>> Joe,
>>> 
>>> I don't have good understanding of KAFKA-1018. Do you think this needs to
>>> be fixed in 0.8 final?
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> 
>>> 
>>> On Fri, Sep 13, 2013 at 9:18 AM, Jun Rao  wrote:
>>> 
 Hi, Everyone,
 
 We have been stabilizing the 0.8 branch since the beta1 release. I think
 we are getting close to an 0.8 final release. I made an initial list of the
 remaining jiras that should be fixed in 0.8.
 
 
 https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%20%220.8%22%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)
 
 1. Do people agree with the list?
 
 2. If the list is good, could people help contributing/reviewing the
 remaining jiras?
 
 Thanks,
 
 Jun
 
>>> 
>>> 
>> 

Re: Review Request 14865: Patch for KAFKA-1097

[Neha Narkhede]

> On Oct. 24, 2013, 4:30 p.m., Jun Rao wrote:
> > Thanks for the patch. There a couple of issues.
> > 
> > 1. The main one is during the phase of partition reassignment when we 
> > bootstrap new replicas. At this point, the assigned replica list doesn't 
> > include the new replicas. If we only allow replicas in assigned replica set 
> > to be added to ISR, those new replicas won't be added to ISR, which will 
> > prevent partition reassignment from completing. We could include those new 
> > replicas in the all replica set in the LeaderAndIsr request. We probably 
> > have to think a bit more to see if there is any other impact.
> > 
> > 2. Once the assigned replica set in the broker is updated. We need to 
> > prevent an old replica from being added back to this set again. Currently, 
> > in Partition.updateLeaderHWAndMaybeExpandIsr() (triggered by a fetch 
> > request), it will call getOrCreateReplica(), which can cause a replica to 
> > be added back to the assigned replica set. What we can do is to only call 
> > getOrCreateReplica during makeLeader() and makeFollower(). In the former, 
> > we force all replicas to be created. In the latter, we just need to make 
> > sure the local replica is created. In 
> > Partition.updateLeaderHWAndMaybeExpandIsr(), we can then use getReplica(), 
> > instead of getOrCreateReplica().

Those are great points 

1. We can't really include the new replicas in the assigned replicas and send 
it to the leader. The reason is that the request would have the same 
leaderEpoch and the broker will ignore the leader and isr request. Hacking 
around this would require the controller to write the state path with a higher 
leader epoch (even if there is no real change to the partition's state) and 
then send the LeaderAndIsrRequest with a higher leaderEpoch
2. This is a good suggestion. Partition.updateLeaderHWAndMaybeExpandIsr() 
should really be using getReplica(). 


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14865/#review27459
---


On Oct. 23, 2013, 4:34 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14865/
> ---
> 
> (Updated Oct. 23, 2013, 4:34 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1097
> https://issues.apache.org/jira/browse/KAFKA-1097
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1097 Race condition while reassigning low throughput partition leads to 
> incorrect ISR information in zookeeper; The changes include 1) Adding the ISR 
> shrink logic as part of the OfflineReplica -> NonExistentReplica state change 
> 2) Adding a safety check on the broker where it only expands the ISR if the 
> replica is in the assigned replica list 3) Updating the assigned replica list 
> on the broker on every makeLeader request and also on makeFollower request 
> for safety, though that's not strictly required. These changes will ensure 
> that the ISR is shrunk by the controller and the leader has an updated 
> assigned replica list. So even if a replica sends a fetch request after the 
> ISR is shrunk by the controller, the broker will not be able to update the 
> ISR until it receives the next LeaderAndIsrRequest (which notifies it of the 
> latest zkVersion of the partition state path) that also contains the shrunk 
> ISR and assigned replica list. Using that the broker will avoid expanding the 
> ISR i
 f the replica is not present in the new assigned replica list
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 60f3ed4e88b08d0dcf2ea84259ae6dbe9d7c3a2d 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 88d130f55997b72a8590e4cfe92857a7320e70d5 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 212c05d65dcdc147e55f90875bacc940e30342bf 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 03ba60e82cdb3dce100603d615894ede47e4b077 
>   kafka-patch-review.py 82ea9a890fe79aad7d0ea6d33f3e2780e036317c 
> 
> Diff: https://reviews.apache.org/r/14865/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 14865: Patch for KAFKA-1097

[Neha Narkhede]

> On Oct. 24, 2013, 12:30 a.m., Guozhang Wang wrote:
> > kafka-patch-review.py, line 100
> > 
> >
> > Should this be included in this RB?

Actually I wanted to improve the tool to print the branch against which the rb 
is created. But I need to revise the patch, this particular diff to 
kafka-patch-review.py can be ignored


> On Oct. 24, 2013, 12:30 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/cluster/Partition.scala, line 211
> > 
> >
> > Previously we do not update assignedReplicas map upon receiving the 
> > LeaderAndISR request, was that OK?

Yes, since it only changed once when the partition was created and never 
changed after that. The exception is partition reassignment, increasing 
replication factor and deleting topic. So we need to fix it


> On Oct. 24, 2013, 12:30 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/controller/ReplicaStateMachine.scala, line 137
> > 
> >
> > The changes in logic of expanding ISR should have already solved the 
> > problem. Do we need to do this here?

Yes, we need to let the leader know of the shrunk assigned replica list. So we 
need the controller to write the ISR and then send the shrunk assigned replica 
list with the shrunk ISR to the leader.


> On Oct. 24, 2013, 12:30 a.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/controller/ReplicaStateMachine.scala, line 181
> > 
> >
> > Ditto as above.

This is required for correctness. Since the controller is actively trying to 
let the replica to stop, it has to send the stop replica request. It also 
reduces the probability of the replica unnecessarily trying to fetch from the 
leader and re-enter ISR


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14865/#review27433
---


On Oct. 23, 2013, 4:34 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14865/
> ---
> 
> (Updated Oct. 23, 2013, 4:34 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1097
> https://issues.apache.org/jira/browse/KAFKA-1097
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1097 Race condition while reassigning low throughput partition leads to 
> incorrect ISR information in zookeeper; The changes include 1) Adding the ISR 
> shrink logic as part of the OfflineReplica -> NonExistentReplica state change 
> 2) Adding a safety check on the broker where it only expands the ISR if the 
> replica is in the assigned replica list 3) Updating the assigned replica list 
> on the broker on every makeLeader request and also on makeFollower request 
> for safety, though that's not strictly required. These changes will ensure 
> that the ISR is shrunk by the controller and the leader has an updated 
> assigned replica list. So even if a replica sends a fetch request after the 
> ISR is shrunk by the controller, the broker will not be able to update the 
> ISR until it receives the next LeaderAndIsrRequest (which notifies it of the 
> latest zkVersion of the partition state path) that also contains the shrunk 
> ISR and assigned replica list. Using that the broker will avoid expanding the 
> ISR i
 f the replica is not present in the new assigned replica list
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 60f3ed4e88b08d0dcf2ea84259ae6dbe9d7c3a2d 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 88d130f55997b72a8590e4cfe92857a7320e70d5 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala 
> 212c05d65dcdc147e55f90875bacc940e30342bf 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 03ba60e82cdb3dce100603d615894ede47e4b077 
>   kafka-patch-review.py 82ea9a890fe79aad7d0ea6d33f3e2780e036317c 
> 
> Diff: https://reviews.apache.org/r/14865/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (71 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1098  Unit test failure in 0.8.1 related to LogCleaner
https://issues.apache.org/jira/browse/KAFKA-1098
KAFKA-1097  Race condition while reassigning low throughput partition leads to 
incorrect ISR information in zookeeper 
https://issues.apache.org/jira/browse/KAFKA-1097
KAFKA-1093  Log.getOffsetsBefore(t, …) does not return the last confirmed 
offset before t
https://issues.apache.org/jira/browse/KAFKA-1093
KAFKA-1086  Improve GetOffsetShell to find metadata automatically
https://issues.apache.org/jira/browse/KAFKA-1086
KAFKA-1082  zkclient dies after UnknownHostException in zk reconnect
https://issues.apache.org/jira/browse/KAFKA-1082
KAFKA-1079  Liars in PrimitiveApiTest that promise to test api in compression 
mode, but don't do this actually
https://issues.apache.org/jira/browse/KAFKA-1079
KAFKA-1060  Break-down sendTime into responseQueueTime and the real sendTime
https://issues.apache.org/jira/browse/KAFKA-1060
KAFKA-1049  Encoder implementations are required to provide an undocumented 
constructor.
https://issues.apache.org/jira/browse/KAFKA-1049
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1020  Remove getAllReplicasOnBroker from KafkaController
https://issues.apache.org/jira/browse/KAFKA-1020
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-1001  Handle follower transition in batch
https://issues.apache.org/jira/browse/KAFKA-1001
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-918   Change log.retention.hours to be log.retention.mins
https://issues.apache.org/jira/browse/KAFKA-918
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and other kafka specific code from the 
RequestChannel
https://issues.apache.org/jira/browse/KAFKA-745
KAFKA-735   Add looping and JSON output for ConsumerOffsetChecker
https://issues.apache.org/jira/browse/KAFKA-735
KAFKA-717   scala 2.10 build support
https://issues.apache.org/jira/browse/KAFKA-717
KAFKA-686   0.8 Kafka broker should give a better error message when running 
against 0.7 zookeeper
https://issues.apache.org/jira/browse/KAFKA-686
KAFKA-674   Clean Shutdown Testing - Log segments che

[jira] [Commented] (KAFKA-1097) Race condition while reassigning low throughput partition leads to incorrect ISR information in zookeeper

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804616#comment-13804616
 ] 

Neha Narkhede commented on KAFKA-1097:
--

Thinking about it more, my original proposed long term fix (mentioned in my 1st 
comment) is much less hackier and precise compared to the one in the rb. 

> Race condition while reassigning low throughput partition leads to incorrect 
> ISR information in zookeeper 
> --
>
> Key: KAFKA-1097
> URL: https://issues.apache.org/jira/browse/KAFKA-1097
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.8.1
>
> Attachments: KAFKA-1097.patch
>
>
> While moving partitions, the controller moves the old replicas through the 
> following state changes -
> ONLINE -> OFFLINE -> NON_EXISTENT
> During the offline state change, the controller removes the old replica and 
> writes the updated ISR to zookeeper and notifies the leader. Note that it 
> doesn't notify the old replicas to stop fetching from the leader (to be fixed 
> in KAFKA-1032). During the non-existent state change, the controller does not 
> write the updated ISR or replica list to zookeeper. Right after the 
> non-existent state change, the controller writes the new replica list to 
> zookeeper, but does not update the ISR. So an old replica can send a fetch 
> request after the offline state change, essentially letting the leader add it 
> back to the ISR. The problem is that if there is no new data coming in for 
> the partition and the old replica is fully caught up, the leader cannot 
> remove it from the ISR. That lets a non existent replica live in the ISR at 
> least until new data comes in to the partition



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1098) Unit test failure in 0.8.1 related to LogCleaner

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804881#comment-13804881
 ] 

Neha Narkhede commented on KAFKA-1098:
--

[~jvanremoortere] That list of Scala versions is correct.

> Unit test failure in 0.8.1 related to LogCleaner
> 
>
> Key: KAFKA-1098
> URL: https://issues.apache.org/jira/browse/KAFKA-1098
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Jay Kreps
> Attachments: kafka_1098-v1.patch, KAFKA-1098-v2.patch
>
>
> Floor = 0, To = -1
> [2013-10-22 09:39:25,001] ERROR Error in cleaner thread 0: 
> (kafka.log.LogCleaner:103)
> java.lang.IllegalArgumentException: inconsistent range
>   at 
> java.util.concurrent.ConcurrentSkipListMap$SubMap.(ConcurrentSkipListMap.java:2506)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:1984)
>   at kafka.log.Log.logSegments(Log.scala:605)
>   at kafka.log.LogToClean.(LogCleaner.scala:596)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>   at scala.collection.immutable.List.foreach(List.scala:45)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>   at scala.collection.immutable.List.map(List.scala:45)
>   at 
> kafka.log.LogCleaner.kafka$log$LogCleaner$$grabFilthiestLog(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203)
>   at kafka.log.LogCleaner$CleanerThread.run(LogCleaner.scala:189)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1103) Consumer uses two zkclients

[Joel Koshy (JIRA)]

Joel Koshy created KAFKA-1103:
-

 Summary: Consumer uses two zkclients
 Key: KAFKA-1103
 URL: https://issues.apache.org/jira/browse/KAFKA-1103
 Project: Kafka
  Issue Type: Bug
Reporter: Joel Koshy
 Fix For: 0.8.1


.. which is very confusing when debugging consumer logs. I don't remember any 
good reason for this, and we should get rid of the one instantiated in 
ZookeeperTopicEventWatcher if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1104) Consumer uses two zkclients

[Joel Koshy (JIRA)]

Joel Koshy created KAFKA-1104:
-

 Summary: Consumer uses two zkclients
 Key: KAFKA-1104
 URL: https://issues.apache.org/jira/browse/KAFKA-1104
 Project: Kafka
  Issue Type: Bug
Reporter: Joel Koshy
 Fix For: 0.8.1


.. which is very confusing when debugging consumer logs. I don't remember any 
good reason for this, and we should get rid of the one instantiated in 
ZookeeperTopicEventWatcher if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1105) Consumer uses two zkclient instances

[Joel Koshy (JIRA)]

Joel Koshy created KAFKA-1105:
-

 Summary: Consumer uses two zkclient instances
 Key: KAFKA-1105
 URL: https://issues.apache.org/jira/browse/KAFKA-1105
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1
Reporter: Joel Koshy


... which makes it very confusing to debug consumer logs. I don't recall a good 
reason for having the second instance in ZooKeeperTopicEventWatcher - we should 
get rid of that if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1104) Consumer uses two zkclients

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy resolved KAFKA-1104.
---

Resolution: Duplicate

Apologies for the spam - my browser was just hanging when i tried to create 
this.

> Consumer uses two zkclients
> ---
>
> Key: KAFKA-1104
> URL: https://issues.apache.org/jira/browse/KAFKA-1104
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
> Fix For: 0.8.1
>
>
> .. which is very confusing when debugging consumer logs. I don't remember any 
> good reason for this, and we should get rid of the one instantiated in 
> ZookeeperTopicEventWatcher if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1105) Consumer uses two zkclient instances

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy resolved KAFKA-1105.
---

Resolution: Duplicate

Apologies for the spam - my browser was just hanging when i tried to create 
this.

> Consumer uses two zkclient instances
> 
>
> Key: KAFKA-1105
> URL: https://issues.apache.org/jira/browse/KAFKA-1105
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
>Reporter: Joel Koshy
>
> ... which makes it very confusing to debug consumer logs. I don't recall a 
> good reason for having the second instance in ZooKeeperTopicEventWatcher - we 
> should get rid of that if possible.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1098) Unit test failure in 0.8.1 related to LogCleaner

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps updated KAFKA-1098:
-

Attachment: KAFKA-1098-v3.patch

Ack, fixed 2.10 issue. Frustrating. Do we have a single sbt command that can 
compile for all?

Jun--the other two null comparisons are for entries not offsets so == should be 
valid there.

> Unit test failure in 0.8.1 related to LogCleaner
> 
>
> Key: KAFKA-1098
> URL: https://issues.apache.org/jira/browse/KAFKA-1098
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Jay Kreps
> Attachments: kafka_1098-v1.patch, KAFKA-1098-v2.patch, 
> KAFKA-1098-v3.patch
>
>
> Floor = 0, To = -1
> [2013-10-22 09:39:25,001] ERROR Error in cleaner thread 0: 
> (kafka.log.LogCleaner:103)
> java.lang.IllegalArgumentException: inconsistent range
>   at 
> java.util.concurrent.ConcurrentSkipListMap$SubMap.(ConcurrentSkipListMap.java:2506)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:1984)
>   at kafka.log.Log.logSegments(Log.scala:605)
>   at kafka.log.LogToClean.(LogCleaner.scala:596)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>   at scala.collection.immutable.List.foreach(List.scala:45)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>   at scala.collection.immutable.List.map(List.scala:45)
>   at 
> kafka.log.LogCleaner.kafka$log$LogCleaner$$grabFilthiestLog(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203)
>   at kafka.log.LogCleaner$CleanerThread.run(LogCleaner.scala:189)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1098) Unit test failure in 0.8.1 related to LogCleaner

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805013#comment-13805013
 ] 

Jun Rao commented on KAFKA-1098:


+1 for v3. 

To compile in all versions of scala, do the following. It's included in 
README.md
 ./sbt +package

> Unit test failure in 0.8.1 related to LogCleaner
> 
>
> Key: KAFKA-1098
> URL: https://issues.apache.org/jira/browse/KAFKA-1098
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Jay Kreps
> Attachments: kafka_1098-v1.patch, KAFKA-1098-v2.patch, 
> KAFKA-1098-v3.patch
>
>
> Floor = 0, To = -1
> [2013-10-22 09:39:25,001] ERROR Error in cleaner thread 0: 
> (kafka.log.LogCleaner:103)
> java.lang.IllegalArgumentException: inconsistent range
>   at 
> java.util.concurrent.ConcurrentSkipListMap$SubMap.(ConcurrentSkipListMap.java:2506)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:1984)
>   at kafka.log.Log.logSegments(Log.scala:605)
>   at kafka.log.LogToClean.(LogCleaner.scala:596)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>   at scala.collection.immutable.List.foreach(List.scala:45)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>   at scala.collection.immutable.List.map(List.scala:45)
>   at 
> kafka.log.LogCleaner.kafka$log$LogCleaner$$grabFilthiestLog(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203)
>   at kafka.log.LogCleaner$CleanerThread.run(LogCleaner.scala:189)



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1098) Unit test failure in 0.8.1 related to LogCleaner

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-1098.
--

Resolution: Fixed

> Unit test failure in 0.8.1 related to LogCleaner
> 
>
> Key: KAFKA-1098
> URL: https://issues.apache.org/jira/browse/KAFKA-1098
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Jay Kreps
> Attachments: kafka_1098-v1.patch, KAFKA-1098-v2.patch, 
> KAFKA-1098-v3.patch
>
>
> Floor = 0, To = -1
> [2013-10-22 09:39:25,001] ERROR Error in cleaner thread 0: 
> (kafka.log.LogCleaner:103)
> java.lang.IllegalArgumentException: inconsistent range
>   at 
> java.util.concurrent.ConcurrentSkipListMap$SubMap.(ConcurrentSkipListMap.java:2506)
>   at 
> java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:1984)
>   at kafka.log.Log.logSegments(Log.scala:605)
>   at kafka.log.LogToClean.(LogCleaner.scala:596)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$$anonfun$5.apply(LogCleaner.scala:137)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
>   at 
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
>   at scala.collection.immutable.List.foreach(List.scala:45)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
>   at scala.collection.immutable.List.map(List.scala:45)
>   at 
> kafka.log.LogCleaner.kafka$log$LogCleaner$$grabFilthiestLog(LogCleaner.scala:137)
>   at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:203)
>   at kafka.log.LogCleaner$CleanerThread.run(LogCleaner.scala:189)



--
This message was sent by Atlassian JIRA
(v6.1#6144)