Re: Review Request 14013: Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Christopher Freeman]

> On Sept. 8, 2013, 4:56 p.m., Neha Narkhede wrote:
> > core/build.sbt, line 26
> > 
> >
> > Does Scala 2.10 require scalatest 1.9.1? 
> > How about Scala 2.9.x ?

There wasn't a 2.10 build of scalatest version 1.8 so I needed to use one of 
the newer versions with a 2.10 build.  For the other cross build cases, they 
will continue to work with the same version of scalatest that they did 
previously.


> On Sept. 8, 2013, 4:56 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/javaapi/Implicits.scala, line 44
> > 
> >
> > Could you please add a comment above this API explaining why we need to 
> > do this ? The same one you have in ByteBufferMessageSet.scala

sure


> On Sept. 8, 2013, 4:56 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/utils/Pool.scala, line 79
> > 
> >
> > In all other places, we have limited the scope of the JavaConversions 
> > import. Could we also do the same here?

yes


- Christopher


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14013/#review25977
---


On Sept. 7, 2013, 11:35 p.m., Christopher Freeman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14013/
> ---
> 
> (Updated Sept. 7, 2013, 11:35 p.m.)
> 
> 
> Review request for kafka and Neha Narkhede.
> 
> 
> Bugs: KAFKA-1046
> https://issues.apache.org/jira/browse/KAFKA-1046
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> This RB contains necessary changes to the kafka source code to support Scala 
> 2.10.x while still maintaining support for Scala 2.8.x
> 
> The existing cause of source incompatibility between 2.8.x and 2.10.x was 
> primarily due to the use of implicit conversions in the 
> scala.collection.JavaConversions object.  Many of them where deprecated (and 
> replaced with differently named methods) and the deprecated methods where 
> removed in 2.10.  I avoided the source incompatibility by importing the 
> implicits with a wild card.
> 
> Scala annotations moved to a different package in 2.9 and the reference to 
> the old location was removed in 2.10.  I couldn't think of a source 
> compatible way to support both in Annotations.scala so since this source file 
> is very small, I created two copies and configured sbt to use the one 
> appropriate for when building for a particular Scala version.
> 
> I also typed many of the catch block case statements with :Throwable.  This 
> change isn't required but starting with Scala 2.9, the compiler will emit 
> warnings if the type is emitted.  Otherwise "case e =>" and case "e: 
> Throwable =>" are equivalent.
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   core/build.sbt c54cf44 
>   core/src/main/scala/kafka/Kafka.scala dafb1ee 
>   core/src/main/scala/kafka/admin/AddPartitionsCommand.scala 5757c32 
>   core/src/main/scala/kafka/admin/AdminUtils.scala c399bc7 
>   core/src/main/scala/kafka/admin/CreateTopicCommand.scala 21c1186 
>   core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 3da4518 
>   core/src/main/scala/kafka/admin/ListTopicCommand.scala c760cc0 
>   core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
> d5de5f3 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala aa61fa1 
>   core/src/main/scala/kafka/client/ClientUtils.scala cc526ec 
>   core/src/main/scala/kafka/cluster/Broker.scala b03dea2 
>   core/src/main/scala/kafka/consumer/ConsoleConsumer.scala 719beb5 
>   core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala fa6b213 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 4395fe3 
>   core/src/main/scala/kafka/consumer/TopicCount.scala c8e8406 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala e7a692a 
>   core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala df83baa 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala ed1ce0b 
>   core/src/main/scala/kafka/controller/KafkaController.scala ab18b7a 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala a084830 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala c964857 
>   core/src/main/scala/kafka/javaapi/FetchRequest.scala b475240 
>   core/src/main/scala/kafka/javaapi/Implicits.scala ee0a71d 
>   core/src/main/scala/kafka/javaapi/OffsetRequest.scala 1c77ff8 
>   core/src/main/scala/kafka/javaapi/TopicMetadata.scala 97b6dcd 
>   core/src/main/scala/kafka/javaapi/TopicMetadataRequest.scala 5f80df7 
>   core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConn

[jira] [Commented] (KAFKA-1050) Support for "no data loss" mode

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761970#comment-13761970
 ] 

Neha Narkhede commented on KAFKA-1050:
--

For 2, we have KAFKA-1028 filed and I had some initial work going on it. Let me 
clean it up and upload a patch.

>>  Does the leader count in "request.required.acks"?

Yes.



> Support for "no data loss" mode
> ---
>
> Key: KAFKA-1050
> URL: https://issues.apache.org/jira/browse/KAFKA-1050
> Project: Kafka
>  Issue Type: Task
>Reporter: Justin SB
>
> I'd love to use Apache Kafka, but for my application data loss is not 
> acceptable.  Even at the expense of availability (i.e. I need C not A in CAP).
> I think there are two things that I need to change to get a quorum model:
> 1) Make sure I set request.required.acks to 2 (for a 3 node cluster) or 3 
> (for a 5 node cluster) on every request, so that I can only write if a quorum 
> is active.
> 2) Prevent the behaviour where a non-ISR can become the leader if all ISRs 
> die.  I think this is as easy as tweaking 
> core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala, 
> essentially to throw an exception around line 64 in the "data loss" case.
> I haven't yet implemented / tested this.  I'd love to get some input from the 
> Kafka-experts on whether my plan is:
>  (a) correct - will this work?
>  (b) complete - have I missed any cases?
>  (c) recommended - is this a terrible idea :-)
> Thanks for any pointers!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request 14013: Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14013/#review25992
---

Ship it!


Ship It!

- Neha Narkhede


On Sept. 9, 2013, 12:40 a.m., Christopher Freeman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14013/
> ---
> 
> (Updated Sept. 9, 2013, 12:40 a.m.)
> 
> 
> Review request for kafka and Neha Narkhede.
> 
> 
> Bugs: KAFKA-1046
> https://issues.apache.org/jira/browse/KAFKA-1046
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> This RB contains necessary changes to the kafka source code to support Scala 
> 2.10.x while still maintaining support for Scala 2.8.x
> 
> The existing cause of source incompatibility between 2.8.x and 2.10.x was 
> primarily due to the use of implicit conversions in the 
> scala.collection.JavaConversions object.  Many of them where deprecated (and 
> replaced with differently named methods) and the deprecated methods where 
> removed in 2.10.  I avoided the source incompatibility by importing the 
> implicits with a wild card.
> 
> Scala annotations moved to a different package in 2.9 and the reference to 
> the old location was removed in 2.10.  I couldn't think of a source 
> compatible way to support both in Annotations.scala so since this source file 
> is very small, I created two copies and configured sbt to use the one 
> appropriate for when building for a particular Scala version.
> 
> I also typed many of the catch block case statements with :Throwable.  This 
> change isn't required but starting with Scala 2.9, the compiler will emit 
> warnings if the type is emitted.  Otherwise "case e =>" and case "e: 
> Throwable =>" are equivalent.
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   core/build.sbt c54cf44 
>   core/src/main/scala/kafka/Kafka.scala dafb1ee 
>   core/src/main/scala/kafka/admin/AddPartitionsCommand.scala 5757c32 
>   core/src/main/scala/kafka/admin/AdminUtils.scala c399bc7 
>   core/src/main/scala/kafka/admin/CreateTopicCommand.scala 21c1186 
>   core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 3da4518 
>   core/src/main/scala/kafka/admin/ListTopicCommand.scala c760cc0 
>   core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
> d5de5f3 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala aa61fa1 
>   core/src/main/scala/kafka/client/ClientUtils.scala cc526ec 
>   core/src/main/scala/kafka/cluster/Broker.scala b03dea2 
>   core/src/main/scala/kafka/consumer/ConsoleConsumer.scala 719beb5 
>   core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala fa6b213 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 4395fe3 
>   core/src/main/scala/kafka/consumer/TopicCount.scala c8e8406 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala e7a692a 
>   core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala df83baa 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala ed1ce0b 
>   core/src/main/scala/kafka/controller/KafkaController.scala ab18b7a 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala a084830 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala c964857 
>   core/src/main/scala/kafka/javaapi/FetchRequest.scala b475240 
>   core/src/main/scala/kafka/javaapi/Implicits.scala ee0a71d 
>   core/src/main/scala/kafka/javaapi/OffsetRequest.scala 1c77ff8 
>   core/src/main/scala/kafka/javaapi/TopicMetadata.scala 97b6dcd 
>   core/src/main/scala/kafka/javaapi/TopicMetadataRequest.scala 5f80df7 
>   core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 
> 14c4c8a 
>   core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
> 0a95248 
>   core/src/main/scala/kafka/javaapi/producer/Producer.scala 7265328 
>   core/src/main/scala/kafka/log/LogManager.scala 4771d11 
>   core/src/main/scala/kafka/network/BoundedByteBufferReceive.scala cab1864 
>   core/src/main/scala/kafka/producer/SyncProducer.scala 306f200 
>   core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala 2e36d3b 
>   core/src/main/scala/kafka/producer/async/ProducerSendThread.scala 2b41a49 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala d5addb3 
>   core/src/main/scala/kafka/server/KafkaApis.scala cd02aab 
>   core/src/main/scala/kafka/server/KafkaServerStartable.scala 5be65e9 
>   core/src/main/scala/kafka/server/ReplicaManager.scala f551243 
>   core/src/main/scala/kafka/server/ZookeeperLeaderElector.scala f1f0625 
>   core/src/main/scala/kafka/tools/ImportZkOffsets.scala 55709b5 
>   core/src/main/scala/kafka/tools/JmxTool.scala 7e424e7 
>   core/src/main/scala/kafka/tools/MirrorMaker.scala 6fb545a 
>   core/src/main/scala/kafka/tools/SimpleConsumerShel

[jira] [Work started] (KAFKA-1028) per topic configuration of preference for consistency over availability

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-1028 started by Neha Narkhede.

> per topic configuration of preference for consistency over availability
> ---
>
> Key: KAFKA-1028
> URL: https://issues.apache.org/jira/browse/KAFKA-1028
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Scott Clasen
>Assignee: Neha Narkhede
>
> As discussed with Neha on the ML.
> It should be possible to configure a topic to disallow unclean leader 
> election, thus preventing the situation where committed messages can be 
> discarded once a failed leader comes back online in a situation where it was 
> the only ISR.
> This would open kafka to additional usecases where the possibility of 
> committted messages being discarded is unacceptable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1046:
-

Attachment: Screen Shot 2013-09-09 at 9.34.09 AM.png

Intellij throws errors while compiling Annotations_2.9+.scala.

> Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
> --
>
> Key: KAFKA-1046
> URL: https://issues.apache.org/jira/browse/KAFKA-1046
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8
>Reporter: Christopher Freeman
>Assignee: Christopher Freeman
> Attachments: kafka_2_10_refactor_0.8.patch, 
> kafka_2_10_refactor.patch, Screen Shot 2013-09-09 at 9.34.09 AM.png
>
>
> I refactored the project such that it will compile against Scala 2.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request 14013: Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Neha Narkhede]

> On Sept. 8, 2013, 10:47 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/utils/Annotations_2.9+.scala, lines 1-38
> > 
> >
> > How does this affect IDEs like Intellij? Are you able to build the 
> > project in IDE with both version of threadsafe annotation?
> 
> Christopher Freeman wrote:
> I don't normally build with my IDE but I don't see any errors when I 
> compile the project with Intellij (Build -> Make project, which invokes an 
> external scala compiler).  I also didn't see any issues using the SBT plugin 
> (which makes sense as all it is doing is opening a shell and running sbt).

I tried compiling the code in IntelliJ and it complains about the some errors 
while compiling Annotations_2.9+.scala. I attached the screenshot on the JIRA - 
https://issues.apache.org/jira/secure/attachment/12602162/Screen%20Shot%202013-09-09%20at%209.34.09%20AM.png.
 Do you know of a way to fix this?


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14013/#review25979
---


On Sept. 9, 2013, 12:40 a.m., Christopher Freeman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14013/
> ---
> 
> (Updated Sept. 9, 2013, 12:40 a.m.)
> 
> 
> Review request for kafka and Neha Narkhede.
> 
> 
> Bugs: KAFKA-1046
> https://issues.apache.org/jira/browse/KAFKA-1046
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> This RB contains necessary changes to the kafka source code to support Scala 
> 2.10.x while still maintaining support for Scala 2.8.x
> 
> The existing cause of source incompatibility between 2.8.x and 2.10.x was 
> primarily due to the use of implicit conversions in the 
> scala.collection.JavaConversions object.  Many of them where deprecated (and 
> replaced with differently named methods) and the deprecated methods where 
> removed in 2.10.  I avoided the source incompatibility by importing the 
> implicits with a wild card.
> 
> Scala annotations moved to a different package in 2.9 and the reference to 
> the old location was removed in 2.10.  I couldn't think of a source 
> compatible way to support both in Annotations.scala so since this source file 
> is very small, I created two copies and configured sbt to use the one 
> appropriate for when building for a particular Scala version.
> 
> I also typed many of the catch block case statements with :Throwable.  This 
> change isn't required but starting with Scala 2.9, the compiler will emit 
> warnings if the type is emitted.  Otherwise "case e =>" and case "e: 
> Throwable =>" are equivalent.
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   core/build.sbt c54cf44 
>   core/src/main/scala/kafka/Kafka.scala dafb1ee 
>   core/src/main/scala/kafka/admin/AddPartitionsCommand.scala 5757c32 
>   core/src/main/scala/kafka/admin/AdminUtils.scala c399bc7 
>   core/src/main/scala/kafka/admin/CreateTopicCommand.scala 21c1186 
>   core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 3da4518 
>   core/src/main/scala/kafka/admin/ListTopicCommand.scala c760cc0 
>   core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
> d5de5f3 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala aa61fa1 
>   core/src/main/scala/kafka/client/ClientUtils.scala cc526ec 
>   core/src/main/scala/kafka/cluster/Broker.scala b03dea2 
>   core/src/main/scala/kafka/consumer/ConsoleConsumer.scala 719beb5 
>   core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala fa6b213 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 4395fe3 
>   core/src/main/scala/kafka/consumer/TopicCount.scala c8e8406 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala e7a692a 
>   core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala df83baa 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala ed1ce0b 
>   core/src/main/scala/kafka/controller/KafkaController.scala ab18b7a 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala a084830 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala c964857 
>   core/src/main/scala/kafka/javaapi/FetchRequest.scala b475240 
>   core/src/main/scala/kafka/javaapi/Implicits.scala ee0a71d 
>   core/src/main/scala/kafka/javaapi/OffsetRequest.scala 1c77ff8 
>   core/src/main/scala/kafka/javaapi/TopicMetadata.scala 97b6dcd 
>   core/src/main/scala/kafka/javaapi/TopicMetadataRequest.scala 5f80df7 
>   core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 
> 14c4c8a 
>   core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
> 0a95248 
>   core/src/main/scala/kafka/javaapi/producer/Producer.scala 726532

[jira] [Commented] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762052#comment-13762052
 ] 

Timothy Chen commented on KAFKA-1036:
-

Hi Jun, I just realized I don't have clearance to provide a patch yet. It will 
be much easier if you can help fix this since it's just a one line fix.

> Unable to rename replication offset checkpoint in windows
> -
>
> Key: KAFKA-1036
> URL: https://issues.apache.org/jira/browse/KAFKA-1036
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: windows
>Reporter: Timothy Chen
>Priority: Critical
>  Labels: windows
>
> Although there was a fix for checkpoint file renaming in windows that tries 
> to delete the existing checkpoint file if renamed failed, I'm still seeing 
> renaming errors on windows even though the destination file doesn't exist.
> A bit investigation shows that it wasn't able to rename the file since the 
> kafka jvm still holds a fie lock on the tmp file and wasn't able to rename 
> it. 
> Attaching a patch that calls a explict writer.close so it can release the 
> lock and can able to rename it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Patch for mmap + windows

[Timothy Chen]

Btw, I've been running this patch in our cloud env and it's been working
fine so far.

I actually filed another bug as I saw another problem on windows locally (
https://issues.apache.org/jira/browse/KAFKA-1036).

Tim


On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps  wrote:

> That would be great!
>
> -Jay
>
>
> On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen  wrote:
>
> > Hi Jay,
> >
> > I'm planning to test run Kafka on Windows in our test environments
> > evaluating if it's suitable for production usage.
> >
> > I can provide feedback with the patch how well it works and if we
> encounter
> > any functional or perf problems.
> >
> > Tim
> >
> >
> >
> >
> > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps  wrote:
> >
> > > Elizabeth and I have a patch to support our memory mapped offset index
> > > files properly on Windows:
> > > https://issues.apache.org/jira/browse/KAFKA-1008
> > >
> > > Question: Do we want this on 0.8 or trunk? I would feel more
> comfortable
> > > with it in trunk, but that means windows support in 0.8 is known to be
> > > broken (as opposed to "not known to be broken but not known to be
> working
> > > either" since we are not doing aggressive system testing on windows).
> > >
> > > I would feel more comfortable doing the patch on 0.8 if there was
> someone
> > > who would be willing to take on real load testing and/or production
> > > operation on Windows so we could have some confidence that Kafka on
> > Windows
> > > actually works, otherwise this could just be the tip of the iceberg.
> > >
> > > Also it would be great to get review on that patch regardless of the
> > > destination.
> > >
> > > -Jay
> > >
> >
>

Re: Patch for mmap + windows

[Jay Kreps]

So guys, do we want to do these in 0.8? The first patch was a little
involved but I think it would be good to have windows support in 0.8 and it
sounds like Tim is able to get things working after these changes.

-Jay


On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen  wrote:

> Btw, I've been running this patch in our cloud env and it's been working
> fine so far.
>
> I actually filed another bug as I saw another problem on windows locally (
> https://issues.apache.org/jira/browse/KAFKA-1036).
>
> Tim
>
>
> On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps  wrote:
>
> > That would be great!
> >
> > -Jay
> >
> >
> > On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen  wrote:
> >
> > > Hi Jay,
> > >
> > > I'm planning to test run Kafka on Windows in our test environments
> > > evaluating if it's suitable for production usage.
> > >
> > > I can provide feedback with the patch how well it works and if we
> > encounter
> > > any functional or perf problems.
> > >
> > > Tim
> > >
> > >
> > >
> > >
> > > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps 
> wrote:
> > >
> > > > Elizabeth and I have a patch to support our memory mapped offset
> index
> > > > files properly on Windows:
> > > > https://issues.apache.org/jira/browse/KAFKA-1008
> > > >
> > > > Question: Do we want this on 0.8 or trunk? I would feel more
> > comfortable
> > > > with it in trunk, but that means windows support in 0.8 is known to
> be
> > > > broken (as opposed to "not known to be broken but not known to be
> > working
> > > > either" since we are not doing aggressive system testing on windows).
> > > >
> > > > I would feel more comfortable doing the patch on 0.8 if there was
> > someone
> > > > who would be willing to take on real load testing and/or production
> > > > operation on Windows so we could have some confidence that Kafka on
> > > Windows
> > > > actually works, otherwise this could just be the tip of the iceberg.
> > > >
> > > > Also it would be great to get review on that patch regardless of the
> > > > destination.
> > > >
> > > > -Jay
> > > >
> > >
> >
>

Re: Review Request 14013: Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Neha Narkhede]

> On Sept. 8, 2013, 4:56 p.m., Neha Narkhede wrote:
> > core/build.sbt, line 26
> > 
> >
> > Does Scala 2.10 require scalatest 1.9.1? 
> > How about Scala 2.9.x ?
> 
> Christopher Freeman wrote:
> There wasn't a 2.10 build of scalatest version 1.8 so I needed to use one 
> of the newer versions with a 2.10 build.  For the other cross build cases, 
> they will continue to work with the same version of scalatest that they did 
> previously.

Makes sense. Thanks!


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14013/#review25977
---


On Sept. 9, 2013, 12:40 a.m., Christopher Freeman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14013/
> ---
> 
> (Updated Sept. 9, 2013, 12:40 a.m.)
> 
> 
> Review request for kafka and Neha Narkhede.
> 
> 
> Bugs: KAFKA-1046
> https://issues.apache.org/jira/browse/KAFKA-1046
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> This RB contains necessary changes to the kafka source code to support Scala 
> 2.10.x while still maintaining support for Scala 2.8.x
> 
> The existing cause of source incompatibility between 2.8.x and 2.10.x was 
> primarily due to the use of implicit conversions in the 
> scala.collection.JavaConversions object.  Many of them where deprecated (and 
> replaced with differently named methods) and the deprecated methods where 
> removed in 2.10.  I avoided the source incompatibility by importing the 
> implicits with a wild card.
> 
> Scala annotations moved to a different package in 2.9 and the reference to 
> the old location was removed in 2.10.  I couldn't think of a source 
> compatible way to support both in Annotations.scala so since this source file 
> is very small, I created two copies and configured sbt to use the one 
> appropriate for when building for a particular Scala version.
> 
> I also typed many of the catch block case statements with :Throwable.  This 
> change isn't required but starting with Scala 2.9, the compiler will emit 
> warnings if the type is emitted.  Otherwise "case e =>" and case "e: 
> Throwable =>" are equivalent.
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   core/build.sbt c54cf44 
>   core/src/main/scala/kafka/Kafka.scala dafb1ee 
>   core/src/main/scala/kafka/admin/AddPartitionsCommand.scala 5757c32 
>   core/src/main/scala/kafka/admin/AdminUtils.scala c399bc7 
>   core/src/main/scala/kafka/admin/CreateTopicCommand.scala 21c1186 
>   core/src/main/scala/kafka/admin/DeleteTopicCommand.scala 3da4518 
>   core/src/main/scala/kafka/admin/ListTopicCommand.scala c760cc0 
>   core/src/main/scala/kafka/admin/PreferredReplicaLeaderElectionCommand.scala 
> d5de5f3 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala aa61fa1 
>   core/src/main/scala/kafka/client/ClientUtils.scala cc526ec 
>   core/src/main/scala/kafka/cluster/Broker.scala b03dea2 
>   core/src/main/scala/kafka/consumer/ConsoleConsumer.scala 719beb5 
>   core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala fa6b213 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 4395fe3 
>   core/src/main/scala/kafka/consumer/TopicCount.scala c8e8406 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala e7a692a 
>   core/src/main/scala/kafka/consumer/ZookeeperTopicEventWatcher.scala df83baa 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala ed1ce0b 
>   core/src/main/scala/kafka/controller/KafkaController.scala ab18b7a 
>   core/src/main/scala/kafka/controller/PartitionStateMachine.scala a084830 
>   core/src/main/scala/kafka/controller/ReplicaStateMachine.scala c964857 
>   core/src/main/scala/kafka/javaapi/FetchRequest.scala b475240 
>   core/src/main/scala/kafka/javaapi/Implicits.scala ee0a71d 
>   core/src/main/scala/kafka/javaapi/OffsetRequest.scala 1c77ff8 
>   core/src/main/scala/kafka/javaapi/TopicMetadata.scala 97b6dcd 
>   core/src/main/scala/kafka/javaapi/TopicMetadataRequest.scala 5f80df7 
>   core/src/main/scala/kafka/javaapi/consumer/ZookeeperConsumerConnector.scala 
> 14c4c8a 
>   core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
> 0a95248 
>   core/src/main/scala/kafka/javaapi/producer/Producer.scala 7265328 
>   core/src/main/scala/kafka/log/LogManager.scala 4771d11 
>   core/src/main/scala/kafka/network/BoundedByteBufferReceive.scala cab1864 
>   core/src/main/scala/kafka/producer/SyncProducer.scala 306f200 
>   core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala 2e36d3b 
>   core/src/main/scala/kafka/producer/async/ProducerSendThread.scala 2b41a49 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala d5add

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (71 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1046  Added support for Scala 2.10 builds while maintaining compatibility 
with 2.8.x
https://issues.apache.org/jira/browse/KAFKA-1046
KAFKA-1042  Fix segment flush logic
https://issues.apache.org/jira/browse/KAFKA-1042
KAFKA-1038  fetch response should use empty messageset instead of null when 
handling errors
https://issues.apache.org/jira/browse/KAFKA-1038
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1020  Remove getAllReplicasOnBroker from KafkaController
https://issues.apache.org/jira/browse/KAFKA-1020
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1008  Unmap before resizing
https://issues.apache.org/jira/browse/KAFKA-1008
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-995   Enforce that the value for replica.fetch.max.bytes is always >= the 
value for message.max.bytes
https://issues.apache.org/jira/browse/KAFKA-995
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-982   Logo for Kafka
https://issues.apache.org/jira/browse/KAFKA-982
KAFKA-981   Unable to pull Kafka binaries with Ivy
https://issues.apache.org/jira/browse/KAFKA-981
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-956   High-level consumer fails to check topic metadata response for 
errors
https://issues.apache.org/jira/browse/KAFKA-956
KAFKA-946   Kafka Hadoop Consumer fails when verifying message checksum
https://issues.apache.org/jira/browse/KAFKA-946
KAFKA-923   Improve controller failover latency
https://issues.apache.org/jira/browse/KAFKA-923
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and other kafka specific code from the 
RequestChannel
https://issues.apache.org/jira/browse/KAFKA-745
KAFKA-735   Add looping and JSON output for ConsumerOffsetChecker
https://issues.apache.org/jira/browse/KAFKA-735
KAFKA-717   scala 2.10 build support
https://issues.apache.org/jira/browse/KAFKA-717
KAFKA-686   0.8 Kafka broker should give a better error message when running 
against 0.7 zookeeper
https://issues.apache.org/jira/browse/KAFKA-686
KAFKA-674   Clean Shutdown Testing - Log segments checksums mismatch
https://issues.apache.org/jira/browse/KAFKA-674
KAFKA-652   Create testcases for clean shut-down
https://issue

[jira] [Commented] (KAFKA-1050) Support for "no data loss" mode

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762070#comment-13762070
 ] 

Jay Kreps commented on KAFKA-1050:
--

Hey Justin, yeah the two things I wanted to clarify:

1. Consider two setups: Kafka with replication factor 2 and 1 failed node, 
zookeeper with replication factor 3 and 1 failed node. Isn't taking writes on 
either of these equally dangerous in the sense that one more failure will leave 
you in an unrecoverable situation? You refer to "majority vote semantics" but I 
think the semantics (disregarding unsafe leader election) are the same, no? Can 
you clarify what you are looking for?

2. Yeah, it's worth clarifying that what we report back is just whether the 
write is just whether the write is "guaranteed". Not guaranteed is not the same 
as "did not occur". So, for example if the client issues a write and then dies 
or becomes partitioned from the cluster the write may succeed even though the 
ack cannot be sent back to the producer. So in the case of the acks=3 with only 
2 available servers we would tell the client "sorry we couldn't get 3x 
replication during the time we waited" but that doesn't mean we guarantee no 
write it just means we got fewer than 3.



 

> Support for "no data loss" mode
> ---
>
> Key: KAFKA-1050
> URL: https://issues.apache.org/jira/browse/KAFKA-1050
> Project: Kafka
>  Issue Type: Task
>Reporter: Justin SB
>
> I'd love to use Apache Kafka, but for my application data loss is not 
> acceptable.  Even at the expense of availability (i.e. I need C not A in CAP).
> I think there are two things that I need to change to get a quorum model:
> 1) Make sure I set request.required.acks to 2 (for a 3 node cluster) or 3 
> (for a 5 node cluster) on every request, so that I can only write if a quorum 
> is active.
> 2) Prevent the behaviour where a non-ISR can become the leader if all ISRs 
> die.  I think this is as easy as tweaking 
> core/src/main/scala/kafka/controller/PartitionLeaderSelector.scala, 
> essentially to throw an exception around line 64 in the "data loss" case.
> I haven't yet implemented / tested this.  I'd love to get some input from the 
> Kafka-experts on whether my plan is:
>  (a) correct - will this work?
>  (b) complete - have I missed any cases?
>  (c) recommended - is this a terrible idea :-)
> Thanks for any pointers!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1049) Encoder implementations are required to provide an undocumented constructor.

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762193#comment-13762193
 ] 

Jay Kreps commented on KAFKA-1049:
--

I agree that an init() method avoids the non-type-checked contructor-type 
requirement, but since serializers are used in a multithreaded context this 
creates difficulties since all the member variables are now mutable and need to 
be synchronized.

> Encoder implementations are required to provide an undocumented constructor.
> 
>
> Key: KAFKA-1049
> URL: https://issues.apache.org/jira/browse/KAFKA-1049
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Rosenberg
>Priority: Minor
>
> So, it seems that if I want to set a custom serializer class on the producer 
> (in 0.8), I have to use a class that includes a special constructor like:
> public class MyKafkaEncoder implements Encoder {
>   // This constructor is expected by the kafka producer, used by reflection
>   public MyKafkaEncoder(VerifiableProperties props) {
> // what can I do with this?
>   }
>  @Override
>   public byte[] toBytes(MyType message) {
> return message.toByteArray();
>   }
> }
> It seems odd that this would be a requirement when implementing an interface. 
>  This seems not to have been the case in 0.7.
> What could my encoder class do with the VerifiableProperties?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-1049) Encoder implementations are required to provide an undocumented constructor.

[Jason Rosenberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762209#comment-13762209
 ] 

Jason Rosenberg commented on KAFKA-1049:


Perhaps it would be cleaner to have the api specify a factory class instead of 
the actual encoder class.  This way, the factory can implement an api which 
requires a getEncoder(Properties props) method.

> Encoder implementations are required to provide an undocumented constructor.
> 
>
> Key: KAFKA-1049
> URL: https://issues.apache.org/jira/browse/KAFKA-1049
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Rosenberg
>Priority: Minor
>
> So, it seems that if I want to set a custom serializer class on the producer 
> (in 0.8), I have to use a class that includes a special constructor like:
> public class MyKafkaEncoder implements Encoder {
>   // This constructor is expected by the kafka producer, used by reflection
>   public MyKafkaEncoder(VerifiableProperties props) {
> // what can I do with this?
>   }
>  @Override
>   public byte[] toBytes(MyType message) {
> return message.toByteArray();
>   }
> }
> It seems odd that this would be a requirement when implementing an interface. 
>  This seems not to have been the case in 0.7.
> What could my encoder class do with the VerifiableProperties?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Review Request 14041: KAFKA-1030

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/
---

Review request for kafka.


Repository: kafka


Description
---

Using the approach of reading directly from ZK.


Diffs
-

  core/src/main/scala/kafka/client/ClientUtils.scala 
cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 

Diff: https://reviews.apache.org/r/14041/diff/


Testing
---

unit tests


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Christopher Freeman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher Freeman updated KAFKA-1046:
---

Attachment: (was: kafka_2_10_refactor_0.8.patch)

> Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
> --
>
> Key: KAFKA-1046
> URL: https://issues.apache.org/jira/browse/KAFKA-1046
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8
>Reporter: Christopher Freeman
>Assignee: Christopher Freeman
> Attachments: kafka_2_10_refactor_0.8.patch, 
> kafka_2_10_refactor.patch, Screen Shot 2013-09-09 at 9.34.09 AM.png
>
>
> I refactored the project such that it will compile against Scala 2.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Christopher Freeman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher Freeman updated KAFKA-1046:
---

Attachment: kafka_2_10_refactor_0.8.patch

> Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
> --
>
> Key: KAFKA-1046
> URL: https://issues.apache.org/jira/browse/KAFKA-1046
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8
>Reporter: Christopher Freeman
>Assignee: Christopher Freeman
> Attachments: kafka_2_10_refactor_0.8.patch, 
> kafka_2_10_refactor.patch, Screen Shot 2013-09-09 at 9.34.09 AM.png
>
>
> I refactored the project such that it will compile against Scala 2.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Patch for mmap + windows

[Neha Narkhede]

+1 for windows support on 0.8

Thanks,
Neha


On Mon, Sep 9, 2013 at 10:48 AM, Jay Kreps  wrote:

> So guys, do we want to do these in 0.8? The first patch was a little
> involved but I think it would be good to have windows support in 0.8 and it
> sounds like Tim is able to get things working after these changes.
>
> -Jay
>
>
> On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen  wrote:
>
> > Btw, I've been running this patch in our cloud env and it's been working
> > fine so far.
> >
> > I actually filed another bug as I saw another problem on windows locally
> (
> > https://issues.apache.org/jira/browse/KAFKA-1036).
> >
> > Tim
> >
> >
> > On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps  wrote:
> >
> > > That would be great!
> > >
> > > -Jay
> > >
> > >
> > > On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen 
> wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > I'm planning to test run Kafka on Windows in our test environments
> > > > evaluating if it's suitable for production usage.
> > > >
> > > > I can provide feedback with the patch how well it works and if we
> > > encounter
> > > > any functional or perf problems.
> > > >
> > > > Tim
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps 
> > wrote:
> > > >
> > > > > Elizabeth and I have a patch to support our memory mapped offset
> > index
> > > > > files properly on Windows:
> > > > > https://issues.apache.org/jira/browse/KAFKA-1008
> > > > >
> > > > > Question: Do we want this on 0.8 or trunk? I would feel more
> > > comfortable
> > > > > with it in trunk, but that means windows support in 0.8 is known to
> > be
> > > > > broken (as opposed to "not known to be broken but not known to be
> > > working
> > > > > either" since we are not doing aggressive system testing on
> windows).
> > > > >
> > > > > I would feel more comfortable doing the patch on 0.8 if there was
> > > someone
> > > > > who would be willing to take on real load testing and/or production
> > > > > operation on Windows so we could have some confidence that Kafka on
> > > > Windows
> > > > > actually works, otherwise this could just be the tip of the
> iceberg.
> > > > >
> > > > > Also it would be great to get review on that patch regardless of
> the
> > > > > destination.
> > > > >
> > > > > -Jay
> > > > >
> > > >
> > >
> >
>

[jira] [Commented] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13762380#comment-13762380
 ] 

Neha Narkhede commented on KAFKA-1046:
--

Chris helped figure out how to get the Intellij build to work. Click on 
Intellij -> Preferences -> Compiler and uncheck "Use external build"

+1 on the updated reviewboard patch.

Chris, will keep this ticket open until you update the trunk patch.

> Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x
> --
>
> Key: KAFKA-1046
> URL: https://issues.apache.org/jira/browse/KAFKA-1046
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8
>Reporter: Christopher Freeman
>Assignee: Christopher Freeman
> Attachments: kafka_2_10_refactor_0.8.patch, 
> kafka_2_10_refactor.patch, Screen Shot 2013-09-09 at 9.34.09 AM.png
>
>
> I refactored the project such that it will compile against Scala 2.10.1.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request 14041: KAFKA-1030

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/
---

(Updated Sept. 9, 2013, 9:30 p.m.)


Review request for kafka.


Repository: kafka


Description
---

Using the approach of reading directly from ZK.


Diffs (updated)
-

  core/src/main/scala/kafka/client/ClientUtils.scala 
cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 

Diff: https://reviews.apache.org/r/14041/diff/


Testing
---

unit tests


Thanks,

Guozhang Wang

Re: Review Request 14041: KAFKA-1030

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/
---

(Updated Sept. 9, 2013, 9:30 p.m.)


Review request for kafka.


Bugs: KAFKA-1030
https://issues.apache.org/jira/browse/KAFKA-1030


Repository: kafka


Description
---

Using the approach of reading directly from ZK.


Diffs
-

  core/src/main/scala/kafka/client/ClientUtils.scala 
cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 

Diff: https://reviews.apache.org/r/14041/diff/


Testing
---

unit tests


Thanks,

Guozhang Wang

Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/
---

(Updated Sept. 9, 2013, 9:31 p.m.)


Review request for kafka.


Summary (updated)
-

MetadataResponse during Consumer's Rebalance Process maybe Stale


Bugs: KAFKA-1030
https://issues.apache.org/jira/browse/KAFKA-1030


Repository: kafka


Description
---

Using the approach of reading directly from ZK.


Diffs
-

  core/src/main/scala/kafka/client/ClientUtils.scala 
cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 

Diff: https://reviews.apache.org/r/14041/diff/


Testing
---

unit tests


Thanks,

Guozhang Wang

Re: review board is available

[Guozhang Wang]

Same concern. It seems that ReviewBoard would not, say, mark the JIRA as
patch available when request is published. Does that mean we have to keep
doing this manually still?

Guozhang


On Sun, Sep 8, 2013 at 10:04 AM, Neha Narkhede wrote:

> This is great. So far I reviewed 2 patches using reviewboard and it is a
> huge time saver. I have one question about the setup though - the
> reviewboard has a Bugs field that takes in the JIRA, can it update the
> respective JIRA with the reviewboard link ?  The concern is that with more
> reviews coming in through review board, it will soon become intractable
> since there is also no easy way to search by Bug in reviewboard.
>
> Thanks,
> Neha
>
>
>
> On Mon, Aug 26, 2013 at 1:00 PM, Jay Kreps  wrote:
>
> > Okay I got review board set up. I'm not wedded to this tool, but it's
> > pretty simple so let's give it a try and if it sucks we can switch to
> > phabricator or something else.
> >
> > I wrote up instructions on how to use it here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Review+Board
> >
> > If anyone has better info on workflow it would be great to help people
> > figure it out. Ideally if we end up liking this tool it would be good to
> > integrate it into the git workflow tool and ideally have some script to
> > handle both the JIRA attachment and the review board.
> >
> > -Jay
> >
>



-- 
-- Guozhang

Re: Patch for mmap + windows

[Jay Kreps]

Cool can we get a reviewer for KAFKA-1008 then? I can take on the other
issue for the checkpoint files.

-Jay


On Mon, Sep 9, 2013 at 3:16 PM, Neha Narkhede wrote:

> +1 for windows support on 0.8
>
> Thanks,
> Neha
>
>
> On Mon, Sep 9, 2013 at 10:48 AM, Jay Kreps  wrote:
>
> > So guys, do we want to do these in 0.8? The first patch was a little
> > involved but I think it would be good to have windows support in 0.8 and
> it
> > sounds like Tim is able to get things working after these changes.
> >
> > -Jay
> >
> >
> > On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen  wrote:
> >
> > > Btw, I've been running this patch in our cloud env and it's been
> working
> > > fine so far.
> > >
> > > I actually filed another bug as I saw another problem on windows
> locally
> > (
> > > https://issues.apache.org/jira/browse/KAFKA-1036).
> > >
> > > Tim
> > >
> > >
> > > On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps 
> wrote:
> > >
> > > > That would be great!
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen 
> > wrote:
> > > >
> > > > > Hi Jay,
> > > > >
> > > > > I'm planning to test run Kafka on Windows in our test environments
> > > > > evaluating if it's suitable for production usage.
> > > > >
> > > > > I can provide feedback with the patch how well it works and if we
> > > > encounter
> > > > > any functional or perf problems.
> > > > >
> > > > > Tim
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps 
> > > wrote:
> > > > >
> > > > > > Elizabeth and I have a patch to support our memory mapped offset
> > > index
> > > > > > files properly on Windows:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1008
> > > > > >
> > > > > > Question: Do we want this on 0.8 or trunk? I would feel more
> > > > comfortable
> > > > > > with it in trunk, but that means windows support in 0.8 is known
> to
> > > be
> > > > > > broken (as opposed to "not known to be broken but not known to be
> > > > working
> > > > > > either" since we are not doing aggressive system testing on
> > windows).
> > > > > >
> > > > > > I would feel more comfortable doing the patch on 0.8 if there was
> > > > someone
> > > > > > who would be willing to take on real load testing and/or
> production
> > > > > > operation on Windows so we could have some confidence that Kafka
> on
> > > > > Windows
> > > > > > actually works, otherwise this could just be the tip of the
> > iceberg.
> > > > > >
> > > > > > Also it would be great to get review on that patch regardless of
> > the
> > > > > > destination.
> > > > > >
> > > > > > -Jay
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Patch for mmap + windows

[Sriram Subramanian]

I did take a look at KAFKA-1008 a while back and added some comments.

On 9/9/13 3:52 PM, "Jay Kreps"  wrote:

>Cool can we get a reviewer for KAFKA-1008 then? I can take on the other
>issue for the checkpoint files.
>
>-Jay
>
>
>On Mon, Sep 9, 2013 at 3:16 PM, Neha Narkhede
>wrote:
>
>> +1 for windows support on 0.8
>>
>> Thanks,
>> Neha
>>
>>
>> On Mon, Sep 9, 2013 at 10:48 AM, Jay Kreps  wrote:
>>
>> > So guys, do we want to do these in 0.8? The first patch was a little
>> > involved but I think it would be good to have windows support in 0.8
>>and
>> it
>> > sounds like Tim is able to get things working after these changes.
>> >
>> > -Jay
>> >
>> >
>> > On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen 
>>wrote:
>> >
>> > > Btw, I've been running this patch in our cloud env and it's been
>> working
>> > > fine so far.
>> > >
>> > > I actually filed another bug as I saw another problem on windows
>> locally
>> > (
>> > > https://issues.apache.org/jira/browse/KAFKA-1036).
>> > >
>> > > Tim
>> > >
>> > >
>> > > On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps 
>> wrote:
>> > >
>> > > > That would be great!
>> > > >
>> > > > -Jay
>> > > >
>> > > >
>> > > > On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen 
>> > wrote:
>> > > >
>> > > > > Hi Jay,
>> > > > >
>> > > > > I'm planning to test run Kafka on Windows in our test
>>environments
>> > > > > evaluating if it's suitable for production usage.
>> > > > >
>> > > > > I can provide feedback with the patch how well it works and if
>>we
>> > > > encounter
>> > > > > any functional or perf problems.
>> > > > >
>> > > > > Tim
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps 
>> > > wrote:
>> > > > >
>> > > > > > Elizabeth and I have a patch to support our memory mapped
>>offset
>> > > index
>> > > > > > files properly on Windows:
>> > > > > > https://issues.apache.org/jira/browse/KAFKA-1008
>> > > > > >
>> > > > > > Question: Do we want this on 0.8 or trunk? I would feel more
>> > > > comfortable
>> > > > > > with it in trunk, but that means windows support in 0.8 is
>>known
>> to
>> > > be
>> > > > > > broken (as opposed to "not known to be broken but not known
>>to be
>> > > > working
>> > > > > > either" since we are not doing aggressive system testing on
>> > windows).
>> > > > > >
>> > > > > > I would feel more comfortable doing the patch on 0.8 if there
>>was
>> > > > someone
>> > > > > > who would be willing to take on real load testing and/or
>> production
>> > > > > > operation on Windows so we could have some confidence that
>>Kafka
>> on
>> > > > > Windows
>> > > > > > actually works, otherwise this could just be the tip of the
>> > iceberg.
>> > > > > >
>> > > > > > Also it would be great to get review on that patch regardless
>>of
>> > the
>> > > > > > destination.
>> > > > > >
>> > > > > > -Jay
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>

Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/#review26005
---



core/src/main/scala/kafka/client/ClientUtils.scala


The concern I have with this change is the potential performance hit to the 
consumer rebalance process. Since it affects the end-to-end message latency, I 
would encourage you to run some performance tests for both a MirrorMaker that 
consumes ~500 topics as well as a consumer that consumes 5 topics, to quantify 
the performance hit.

Also, what do you think about the alternate solution that I described on 
the JIRA?


- Neha Narkhede


On Sept. 9, 2013, 9:31 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14041/
> ---
> 
> (Updated Sept. 9, 2013, 9:31 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1030
> https://issues.apache.org/jira/browse/KAFKA-1030
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Using the approach of reading directly from ZK.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/client/ClientUtils.scala 
> cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 
> 
> Diff: https://reviews.apache.org/r/14041/diff/
> 
> 
> Testing
> ---
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale

[Guozhang Wang]

> On Sept. 9, 2013, 11:44 p.m., Neha Narkhede wrote:
> > core/src/main/scala/kafka/client/ClientUtils.scala, line 100
> > 
> >
> > The concern I have with this change is the potential performance hit to 
> > the consumer rebalance process. Since it affects the end-to-end message 
> > latency, I would encourage you to run some performance tests for both a 
> > MirrorMaker that consumes ~500 topics as well as a consumer that consumes 5 
> > topics, to quantify the performance hit.
> > 
> > Also, what do you think about the alternate solution that I described 
> > on the JIRA?

Yeah I will do the perf test.

About the alternative approach, that would require the handling api to 
recognize if the current broker is controller or not; and if yes, hand over the 
request to the controller module. I think that would complicate the logic. What 
do you think?


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/#review26005
---


On Sept. 9, 2013, 9:31 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14041/
> ---
> 
> (Updated Sept. 9, 2013, 9:31 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1030
> https://issues.apache.org/jira/browse/KAFKA-1030
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Using the approach of reading directly from ZK.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/client/ClientUtils.scala 
> cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 
> 
> Diff: https://reviews.apache.org/r/14041/diff/
> 
> 
> Testing
> ---
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Patch for mmap + windows

[Timothy Chen]

Hi Sriram,

If Elizabeth isn't able to respond to your feedback, I wonder if you can
fix the patch and apply it?

It will be great to have this patch in 0.8.

Thanks,

Tim


On Mon, Sep 9, 2013 at 3:56 PM, Sriram Subramanian <
srsubraman...@linkedin.com> wrote:

> I did take a look at KAFKA-1008 a while back and added some comments.
>
> On 9/9/13 3:52 PM, "Jay Kreps"  wrote:
>
> >Cool can we get a reviewer for KAFKA-1008 then? I can take on the other
> >issue for the checkpoint files.
> >
> >-Jay
> >
> >
> >On Mon, Sep 9, 2013 at 3:16 PM, Neha Narkhede
> >wrote:
> >
> >> +1 for windows support on 0.8
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Mon, Sep 9, 2013 at 10:48 AM, Jay Kreps  wrote:
> >>
> >> > So guys, do we want to do these in 0.8? The first patch was a little
> >> > involved but I think it would be good to have windows support in 0.8
> >>and
> >> it
> >> > sounds like Tim is able to get things working after these changes.
> >> >
> >> > -Jay
> >> >
> >> >
> >> > On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen 
> >>wrote:
> >> >
> >> > > Btw, I've been running this patch in our cloud env and it's been
> >> working
> >> > > fine so far.
> >> > >
> >> > > I actually filed another bug as I saw another problem on windows
> >> locally
> >> > (
> >> > > https://issues.apache.org/jira/browse/KAFKA-1036).
> >> > >
> >> > > Tim
> >> > >
> >> > >
> >> > > On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps 
> >> wrote:
> >> > >
> >> > > > That would be great!
> >> > > >
> >> > > > -Jay
> >> > > >
> >> > > >
> >> > > > On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen 
> >> > wrote:
> >> > > >
> >> > > > > Hi Jay,
> >> > > > >
> >> > > > > I'm planning to test run Kafka on Windows in our test
> >>environments
> >> > > > > evaluating if it's suitable for production usage.
> >> > > > >
> >> > > > > I can provide feedback with the patch how well it works and if
> >>we
> >> > > > encounter
> >> > > > > any functional or perf problems.
> >> > > > >
> >> > > > > Tim
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps  >
> >> > > wrote:
> >> > > > >
> >> > > > > > Elizabeth and I have a patch to support our memory mapped
> >>offset
> >> > > index
> >> > > > > > files properly on Windows:
> >> > > > > > https://issues.apache.org/jira/browse/KAFKA-1008
> >> > > > > >
> >> > > > > > Question: Do we want this on 0.8 or trunk? I would feel more
> >> > > > comfortable
> >> > > > > > with it in trunk, but that means windows support in 0.8 is
> >>known
> >> to
> >> > > be
> >> > > > > > broken (as opposed to "not known to be broken but not known
> >>to be
> >> > > > working
> >> > > > > > either" since we are not doing aggressive system testing on
> >> > windows).
> >> > > > > >
> >> > > > > > I would feel more comfortable doing the patch on 0.8 if there
> >>was
> >> > > > someone
> >> > > > > > who would be willing to take on real load testing and/or
> >> production
> >> > > > > > operation on Windows so we could have some confidence that
> >>Kafka
> >> on
> >> > > > > Windows
> >> > > > > > actually works, otherwise this could just be the tip of the
> >> > iceberg.
> >> > > > > >
> >> > > > > > Also it would be great to get review on that patch regardless
> >>of
> >> > the
> >> > > > > > destination.
> >> > > > > >
> >> > > > > > -Jay
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
>

Re: Patch for mmap + windows

[Jay Kreps]

I think Srirams complaint is that I haven't yet addressed his concerns :-)

Sent from my iPhone

On Sep 9, 2013, at 3:56 PM, Sriram Subramanian  
wrote:

> I did take a look at KAFKA-1008 a while back and added some comments.
> 
> On 9/9/13 3:52 PM, "Jay Kreps"  wrote:
> 
>> Cool can we get a reviewer for KAFKA-1008 then? I can take on the other
>> issue for the checkpoint files.
>> 
>> -Jay
>> 
>> 
>> On Mon, Sep 9, 2013 at 3:16 PM, Neha Narkhede
>> wrote:
>> 
>>> +1 for windows support on 0.8
>>> 
>>> Thanks,
>>> Neha
>>> 
>>> 
>>> On Mon, Sep 9, 2013 at 10:48 AM, Jay Kreps  wrote:
>>> 
 So guys, do we want to do these in 0.8? The first patch was a little
 involved but I think it would be good to have windows support in 0.8
>>> and
>>> it
 sounds like Tim is able to get things working after these changes.
 
 -Jay
 
 
 On Mon, Sep 9, 2013 at 10:19 AM, Timothy Chen 
>>> wrote:
 
> Btw, I've been running this patch in our cloud env and it's been
>>> working
> fine so far.
> 
> I actually filed another bug as I saw another problem on windows
>>> locally
 (
> https://issues.apache.org/jira/browse/KAFKA-1036).
> 
> Tim
> 
> 
> On Wed, Aug 21, 2013 at 4:29 PM, Jay Kreps 
>>> wrote:
> 
>> That would be great!
>> 
>> -Jay
>> 
>> 
>> On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen 
 wrote:
>> 
>>> Hi Jay,
>>> 
>>> I'm planning to test run Kafka on Windows in our test
>>> environments
>>> evaluating if it's suitable for production usage.
>>> 
>>> I can provide feedback with the patch how well it works and if
>>> we
>> encounter
>>> any functional or perf problems.
>>> 
>>> Tim
>>> 
>>> 
>>> 
>>> 
>>> On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps 
> wrote:
>>> 
 Elizabeth and I have a patch to support our memory mapped
>>> offset
> index
 files properly on Windows:
 https://issues.apache.org/jira/browse/KAFKA-1008
 
 Question: Do we want this on 0.8 or trunk? I would feel more
>> comfortable
 with it in trunk, but that means windows support in 0.8 is
>>> known
>>> to
> be
 broken (as opposed to "not known to be broken but not known
>>> to be
>> working
 either" since we are not doing aggressive system testing on
 windows).
 
 I would feel more comfortable doing the patch on 0.8 if there
>>> was
>> someone
 who would be willing to take on real load testing and/or
>>> production
 operation on Windows so we could have some confidence that
>>> Kafka
>>> on
>>> Windows
 actually works, otherwise this could just be the tip of the
 iceberg.
 
 Also it would be great to get review on that patch regardless
>>> of
 the
 destination.
 
 -Jay
> 

Re: review board is available

[Jay Kreps]

Yeah the lack of integration is kind of maddening if you have ever used
github.

My recommendation is that we treat Review Board as just a patch viewer and
discussion tool. In other words you must have a JIRA and a link to the
appropriate review board there. We should make no effort to use review
board's dashboard or close the rbs or anything like that. I think it is
reasonable to give the +1 in RB, though.

Not sure how other projects do it.

One simple thing that would help would be to get a wrapper script for
post-review and the jira command-line tool (
https://bobswift.atlassian.net/wiki/display/JCLI/JIRA+Command+Line+Interface).
I want something that takes a JIRA such as KAFKA-123 and generates a patch
and uploads it to both JIRA and rb, updating an existing rb if one is
specified.

-Jay


On Mon, Sep 9, 2013 at 2:32 PM, Guozhang Wang  wrote:

> Same concern. It seems that ReviewBoard would not, say, mark the JIRA as
> patch available when request is published. Does that mean we have to keep
> doing this manually still?
>
> Guozhang
>
>
> On Sun, Sep 8, 2013 at 10:04 AM, Neha Narkhede  >wrote:
>
> > This is great. So far I reviewed 2 patches using reviewboard and it is a
> > huge time saver. I have one question about the setup though - the
> > reviewboard has a Bugs field that takes in the JIRA, can it update the
> > respective JIRA with the reviewboard link ?  The concern is that with
> more
> > reviews coming in through review board, it will soon become intractable
> > since there is also no easy way to search by Bug in reviewboard.
> >
> > Thanks,
> > Neha
> >
> >
> >
> > On Mon, Aug 26, 2013 at 1:00 PM, Jay Kreps  wrote:
> >
> > > Okay I got review board set up. I'm not wedded to this tool, but it's
> > > pretty simple so let's give it a try and if it sucks we can switch to
> > > phabricator or something else.
> > >
> > > I wrote up instructions on how to use it here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Review+Board
> > >
> > > If anyone has better info on workflow it would be great to help people
> > > figure it out. Ideally if we end up liking this tool it would be good
> to
> > > integrate it into the git workflow tool and ideally have some script to
> > > handle both the JIRA attachment and the review board.
> > >
> > > -Jay
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale

[Swapnil Ghike]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/#review26012
---



core/src/main/scala/kafka/client/ClientUtils.scala


You can reuse ZkUtils.getPartitionAssignmentForTopics instead of writing a 
new function.


- Swapnil Ghike


On Sept. 9, 2013, 9:31 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14041/
> ---
> 
> (Updated Sept. 9, 2013, 9:31 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1030
> https://issues.apache.org/jira/browse/KAFKA-1030
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Using the approach of reading directly from ZK.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/client/ClientUtils.scala 
> cc526ec933052b239f0e7ce43e76cd9d011d5bd9 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 
> 
> Diff: https://reviews.apache.org/r/14041/diff/
> 
> 
> Testing
> ---
> 
> unit tests
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>