Re: kafka build error scala 2.10

yes it worked thanks. Regards, Pushkar On Thu, Dec 19, 2013 at 10:49 AM, Jun Rao wrote: > Could you try just excluding Annotation_2.8.scala? > > Thanks, > > Jun > > > On Wed, Dec 18, 2013 at 8:56 PM, pushkar priyadarshi < > priyadarshi.push...@gmail.com> wrote: > > > i see two files name Annot

Re: kafka build error scala 2.10

Could you try just excluding Annotation_2.8.scala? Thanks, Jun On Wed, Dec 18, 2013 at 8:56 PM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > i see two files name Annotation_2.8.scala and Annotation_2.9.scala. > Excluding them does not help.Is this what you were referring to? >

Re: Writing unit tests for Kafka code

You can take a look at Kafka's unit tests. See how kafka.integration.KafkaServerTestHarness is being used. Thanks, Jun On Wed, Dec 18, 2013 at 11:17 AM, Oliver Dain wrote: > Found some of the other references I'd used that also don't quite work: > > https://gist.github.com/fjavieralba/7930018

Re: Consumer Group Rebalance Issues

The issue is that consumer 007 didn't see consumer 006 during rebalancing. So, it made a decision in conflict with consumer 006. Consumer 007 should have another ZK watcher fired to trigger another rebalance when if it will see consumer 006. Which version of ZK are you using? Thanks, Jun On Wed

Re: Kafka 0.8.0 server-stop.sh does not stop broker

Maybe we should just change it to "kill -15". Could you try if that works? Could you also file a jira? Thanks, Jun On Tue, Dec 17, 2013 at 9:18 PM, Bryan Baugher wrote: > RHEL 6.4 64bit > Java 6u35 > > > On Tue, Dec 17, 2013 at 10:57 PM, Jun Rao wrote: > > > Which OS are you on? > > > > Than

Re: Writing unit tests for Kafka code

Found some of the other references I'd used that also don't quite work: https://gist.github.com/fjavieralba/7930018 https://gist.github.com/mardambey/2650743 I know, the obvious question is what didn't work. I'm afraid I no longer recall. I took each example, tried it, got errors, tried to fix t

Writing unit tests for Kafka code

I'm writing some Kafka client code and I'd like to unit test it. I've found some resources that say this is possible. For example: http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/ but all the information I've found seems a little bit incorrect. For example, the above link us

Re: kafka build error scala 2.10

i see two files name Annotation_2.8.scala and Annotation_2.9.scala. Excluding them does not help.Is this what you were referring to? Regards, Pushkar On Wed, Dec 18, 2013 at 9:52 PM, Jun Rao wrote: > You may have to exclude Annotations.scala. > > Thanks, > > Jun > > > On Wed, Dec 18, 2013 at 1

Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

Yes, you need to build kafka first ./sbt update ./sbt package ./sbt assembly-package-dependency Once you can make it reproducible please make a pull request if you could to that repo and I can run or do whatever you did. The /vagrant directory is shared as the folder that you did the vagrant up

Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

Hi Joe, I'm trying to reproduce it with the Vagrant setup you provided. Thanks for setting that up! I should also need to run the sbt commands from the README to build Kafka, right? You included the output from "bin/kafka-list-topic.sh". Based on the problem I've described, this wouldn't show the

RE: a consumer question

Thanks for confirming that, Guozhang. Regards, Libo -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Wednesday, December 18, 2013 4:34 PM To: users@kafka.apache.org Subject: Re: a consumer question Jun is right. Just checked the code. If you set consumer.timeou

Re: Data loss in case of request.required.acks set to -1

Kafka server's shutdown hook should capture all SIG but NOT SIGKILL (-9), and the controlled shut down process will not be triggered in this case. That said, if the other replica is in ISR then even kill -9 should not lose data. I am currently working on this JIRA that might be related if brokers

Re: a consumer question

Jun is right. Just checked the code. If you set consumer.timeout.ms to 0 then if there is no message a ConsumerTimeoutException will be thrown right away. On Tue, Dec 17, 2013 at 9:08 PM, Jun Rao wrote: > Actually, hasNext() only returns false when the consumer connector is > shutdown. Typicall

RE: a consumer question

Thanks, Jun. That is also my guess:) If the exception is caught, I can easily convert hasNext() from blocking to nonblocking. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, December 18, 2013 12:09 AM To: users@kafka.apache.org Subject: Re: a c

Re: problem with high-level consumer stream filter regex....

thanks Joe! On Wed, Dec 18, 2013 at 11:05 AM, Joe Stein wrote: > Hey Jason, I have someone looking into it now (they just started). > > I can look at it on Friday or if I finish up what I am working on for > tomorrow then sooner. > > /*** > Joe Stein >

Re: Data loss in case of request.required.acks set to -1

leader election should start for the brokers that are in the isr for the partitions that are on that replica that are leaders by the other replicas still in the isr, and the leader failed removed from the isr. The isr will shrink for all other partitions this broker is in the isr on but not the le

Re: Data loss in case of request.required.acks set to -1

what happens if the physical machine dies or the kernel panics? On Dec 18, 2013, at 9:44 AM, Hanish Bansal wrote: > Yup definitely i would like to try that If controlled.shutdown.enable > property works in case of kill -9. > > I hope that this option will be perfect. > > Thanks for quick resp

Re: Data loss in case of request.required.acks set to -1

Yup definitely i would like to try that If controlled.shutdown.enable property works in case of kill -9. I hope that this option will be perfect. Thanks for quick response, really appreciate it. On Wed, Dec 18, 2013 at 10:52 PM, Joe Stein wrote: > Wouldn't you want to set the controlled.shutd

Re: Consumer Group Rebalance Issues

Thanks for the help with this Jun, really appreciate it! So I found this in the logs for consumer 007 about an hour previous. Besides that no real activity. It looks like 007 rebalanced and successfully claimed partition 24-27. Shortly after that its zookeeper client timed out and reconnected.

Re: Data loss in case of request.required.acks set to -1

Wouldn't you want to set the controlled.shutdown.enable to true so the broker would do this for you before ending itself? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Data loss in case of request.required.acks set to -1

I am having 2 replicas. Guozhang, I tried both scenario first is only using kill -9 and 2nd is kill and bouncing that. On Wed, Dec 18, 2013 at 10:06 PM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > my doubt was they are dropping off at producer level only.so suggested > play

Re: Migrating a cluster from 0.8.0 to 0.8.1

Thanks Neha, I rolled upgrades and completed a rebalance! I ran into a few small issues I figured I would share. On a few Brokers, there were some log directories left over from some failed rebalances which prevented the 0.8.1 brokers from starting once I completed the upgrade. These directories

Re: Data loss in case of request.required.acks set to -1

my doubt was they are dropping off at producer level only.so suggested playing with paramaters like retries and backoff.ms and also with refreshinterval on producer side. Regards, Pushkar On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang wrote: > Hanish, > > Did you "kill -9" one of the brokers

Re: regarding run-simulator.sh

Most tools can be found in the kafka.tools package. For dumping jmx, use JmxTool. Thanks, Jun On Wed, Dec 18, 2013 at 1:03 AM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > i see many tools mentioned for perf here > > https://cwiki.apache.org/confluence/display/KAFKA/Performanc

Re: Data loss in case of request.required.acks set to -1

Hanish, Did you "kill -9" one of the brokers only or bouncing them iteratively? Guozhang On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein wrote: > How many replicas do you have? > > > On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal < > hanish.bansal.agar...@gmail.com> wrote: > > > Hi pushkar, > > >

Re: kafka build error scala 2.10

You may have to exclude Annotations.scala. Thanks, Jun On Wed, Dec 18, 2013 at 12:16 AM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > While doing dev setup as described in > https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup > > im getting following build errors

Re: Can we archive our kafka-requestlog in kafka0.8??

Take a look at http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppenderand search for compress. Thanks, Jun On Tue, Dec 17, 2013 at 11:58 PM, Nishant Kumar wrote: > Hello All, > > I am using kafka 0.8. > I want to archive my kafka-request.log, server.log, controller.log,

Re: Data loss in case of request.required.acks set to -1

How many replicas do you have? On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi pushkar, > > I tried with configuring "message.send.max.retries" to 10. Default value > for this is 3. > > But still facing data loss. > > > On Wed, Dec 18, 2013 at 12:44

Re: problem with high-level consumer stream filter regex....

Hey Jason, I have someone looking into it now (they just started). I can look at it on Friday or if I finish up what I am working on for tomorrow then sooner. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stea

Re: Data loss in case of request.required.acks set to -1

Hi pushkar, I tried with configuring "message.send.max.retries" to 10. Default value for this is 3. But still facing data loss. On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > You can try setting a higher value for "message.send.max.retries" in

Re: problem with high-level consumer stream filter regex....

Joe, I think the java code I listed in the Jira ticket should reproduce the issue directly, does that not work? Jason On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein wrote: > Hi Jason, I just replied on the ticket. If it is a bug the update to > create new filter or fix as bug, same. > > Can you

Re: Data loss in case of request.required.acks set to -1

You can try setting a higher value for "message.send.max.retries" in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi All, > > We are having kafka cluster of 2 nodes. (using 0.8.0 final release) > Replication Factor:

Data loss in case of request.required.acks set to -1

Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting th

Re: Kafka producer behavior

Hi, this is a gotcha about kafka producer partitioning, you much send the messages with a non null key. If the key is null kafka will not call the partitioner. Because with this partitioner the key does not matter you can pass in a constant string like "1" etc. Oh one more thing, on performance:

Re: Kafka producer behavior

Thanks for response Gerrit and Guozhang !! Hi Gerrit, I am trying to use same round robin partitioner shared by you but hard luck, still round robin partitioning not working. I have successfully registered RoundRobinPartitioner in kafka producer. Code of RoundRobinPartitioner class as: pu

Re: regarding run-simulator.sh

i see many tools mentioned for perf here https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing of all these what all already exist in 0.8 release? e.g. i was not able to find jmx-dump.sh , R script etc anywhere. On Wed, Dec 18, 2013 at 11:01 AM, pushkar priyadarshi < priyadarshi

kafka build error scala 2.10

While doing dev setup as described in https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup im getting following build errors. immutable is already defined as class immutable Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem threadsafe is alre

Can we archive our kafka-requestlog in kafka0.8??

Hello All, I am using kafka 0.8. I want to archive my kafka-request.log, server.log, controller.log, staet-change.log. Is there any property in log4j by which i can archive my logs in zip folders or files. Please suggest. Regards, Nishant Kumar