Re: new log.dirs property (as opposed to log.dir)

is it possible for a partition to have multiple replicas on different directories on the same broker? (hopefully no!) On Wed, Jun 19, 2013 at 10:47 PM, Jun Rao wrote: > It takes a comma separated list and partition replicas are randomly > distributed to the list. > > Thanks, > > Jun > > > On W

Re: new log.dirs property (as opposed to log.dir)

It takes a comma separated list and partition replicas are randomly distributed to the list. Thanks, Jun On Wed, Jun 19, 2013 at 10:25 PM, Jason Rosenberg wrote: > In the 0.8 config, log.dir is now log.dirs. It looks like the singular > log.dir is still supported, but under the covers the pr

new log.dirs property (as opposed to log.dir)

In the 0.8 config, log.dir is now log.dirs. It looks like the singular log.dir is still supported, but under the covers the property is log.dirs. I'm curious, does this take a comma separated list of directories? The new config page just says: "The directories in which the log data is kept" Als

Re: Questions on kafka leader selection

We do have a tool ReassignPartitionsCommand that can move existing replicas to new brokers. It hasn't been thoroughly tested though. Thanks, Jun On Wed, Jun 19, 2013 at 9:41 PM, Yogesh Sangvikar < yogesh.sangvi...@gmail.com> wrote: > Hi Jun, > > Thanks for the reply. > > But this case arises o

Re: Questions on kafka leader selection

Hi Jun, Thanks for the reply. But this case arises one more questions that, If all the 3 down brokers are crashed and we have added some new brokers, is there any way to re-balance those replicas to make partition available? OR any tool to balance the number of replicas if any one of the ISR goes

Re: question about Controlled Shutdown

Nice! Thanks, Jason On Wed, Jun 19, 2013 at 9:16 PM, Jun Rao wrote: > Actually, we recently added the option to enable controlled shutdown in the > broker shutdown hook. I have updated our wiki accordingly ( > https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools). > > Thanks, >

Re: Kafka stats description

Thanks for response :) Its really helpful. On Thu, Jun 20, 2013 at 3:59 AM, Jun Rao wrote: > TotalFetchRequestMs is probably not important. AvgFetchRequestMs and > MaxFetchRequestMs > are and they are in millisecs. > > Thanks, > > Jun > > > On Wed, Jun 19, 2013 at 9:17 AM, Hanish Bansal < > ha

Re: question about Controlled Shutdown

Actually, we recently added the option to enable controlled shutdown in the broker shutdown hook. I have updated our wiki accordingly ( https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools). Thanks, Jun On Wed, Jun 19, 2013 at 6:32 PM, Jason Rosenberg wrote: > Was just reading

RE: C# client for kafka 0.8

Thanks for the information, Tim. I've never coded on the clr, so I asked only to attempt to entice an app group here, who does use C#. They are still considering the situation. thanks, rob Sent: Tuesday, June 18, 2013 9:47 AM To: users@kafka.apache.org Subject: Re: C# client for kafka 0.8

Re: Partition data for deleted topic found in kafka-logs, also, found leader: -1

We never tried manipulating ZK directly. So it may or may not work. Our plan is to have tools to change # partitions, replication factor, etc in the future. Thanks, Jun On Wed, Jun 19, 2013 at 6:18 PM, Jason Rosenberg wrote: > Is there a way to manually do this? Or is there a way to manually

Re: Kafka stats description

TotalFetchRequestMs is probably not important. AvgFetchRequestMs and MaxFetchRequestMs are and they are in millisecs. Thanks, Jun On Wed, Jun 19, 2013 at 9:17 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi > > If i look at these numbers(getting from kafka) then according to t

question about Controlled Shutdown

Was just reading about Controlled Shutdown here: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools Is this something that can be invoked from code, from within a container running the KafkaServer? Currently I launch kafka.server.KafkaServer directly from our java app container.

Re: Partition data for deleted topic found in kafka-logs, also, found leader: -1

Is there a way to manually do this? Or is there a way to manually change the replication factor for a topic? Possibly by manipulating zookeeper data directly? Jason On Wed, Jun 19, 2013 at 8:17 AM, Jun Rao wrote: > Yogesh, > > Delete topic is not supported in 0.8 beta yet. We decided to take

Re: 0.8 beta already deploy in production environment

Thanks Jun, this helped me a lot. -- Markus Roder Am 19.06.2013 um 17:21 schrieb Jun Rao : > We just completed the deployment of Kafka 0.8 (latest code in 0.8 branch) > to production at LinkedIn yesterday. No major issues were found. Be aware > that since this is new system, it does take a bit

Re: Kafka stats description

Hi If i look at these numbers(getting from kafka) then according to that Ms should be microseconds. NumFetchRequests=5350287 TotalFetchRequestMs=56605349834 FetchRequestsPerSecond=92.76876273667219 *Calculation of **FetchRequestsPerSecond manually:* FetchRequestsPerSecond = NumFetchRequests/

Re: Partition data for deleted topic found in kafka-logs, also, found leader: -1

Yogesh, Delete topic is not supported in 0.8 beta yet. We decided to take this feature out in 0.8 beta to make the release available sooner. Thanks, Jun On Tue, Jun 18, 2013 at 6:25 AM, Yogesh Sangvikar < yogesh.sangvi...@gmail.com> wrote: > Hi Team, > > I am exploring kafka 0.8 beta release

Re: Questions on kafka leader selection

If leader is -1, that indicates the partition is unavailable. Were replicas 4,2,3 all down at that time? If so, you need to start at least one of those brokers to make the partition available again. Thanks, Jun On Wed, Jun 19, 2013 at 5:05 AM, Yogesh Sangvikar < yogesh.sangvi...@gmail.com> wrot

Re: 0.8 beta already deploy in production environment

We just completed the deployment of Kafka 0.8 (latest code in 0.8 branch) to production at LinkedIn yesterday. No major issues were found. Be aware that since this is new system, it does take a bit more effort to operate. See some of the new tools at http://kafka.apache.org/08/tools.html Thanks,

Re: kafka source code run problems

Hi Sphinx, I'm running Kafka in Eclipse, could you please explain better your issue? Best, Andrea On 06/18/2013 09:36 AM, sphinx jiang wrote: > Hi, I am Sphinx Jiang, a Chinese Kafka user. > > I am learning the source code. But after I import the source code in > eclipse, the running come with so

Re: kafka source code run problems

Have you looked at https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup? BTW, Apache mailing list doesn't support attachment. Thanks, Jun On Tue, Jun 18, 2013 at 12:36 AM, sphinx jiang wrote: > Hi, I am Sphinx Jiang, a Chinese Kafka user. > > I am learning the source code. But af

Re: kafka service delete the topic directory unexpectedly

Any error/exception on the broker? Thanks, Jun On Mon, Jun 17, 2013 at 10:36 PM, panfei wrote: > we are using kafka-0.7.2 and deploy it on CentOS 6.4 Final 64bit. > > > 2013/6/18 Jun Rao > > > Which version of Kafka are you using? Are you on Windows? > > > > Thanks, > > > > Jun > > > > > > O

Questions on kafka leader selection

Hi Team, Please help to understand following questing as we are planning to use kafka 0.8 in our production, 1. We found the 'leader: -1' state, when the leader selection will happen for such partitions? Does the partition data will be unavailable till that time? example: topic: test1partitio