Re: Recommended log level in prod environment.

Yes, the kafka-request log logs every request (in TRACE). It's mostly for debugging purpose. Other than that, there is no harm to turn it off. Thanks, Jun On Mon, Jul 22, 2013 at 7:59 PM, Calvin Lei wrote: > nah. We just changed it to INFO and will monitor the log. We have GBs of > logs when

Re: Replacing brokers in a cluster (0.8)

You can try kafka-reassign-partitions now. You do have to specify the new replica assignment manually. We are improving that tool to make it more automatic. Thanks, Jun On Mon, Jul 22, 2013 at 10:40 AM, Jason Rosenberg wrote: > Is the kafka-reassign-partitions tool something I can experiment

Re: Recommended log level in prod environment.

nah. We just changed it to INFO and will monitor the log. We have GBs of logs when it was at trace level. the kafka-request log was going crazy. On Jul 22, 2013, at 10:54 PM, Jay Kreps wrote: > We run at info too except when debugging stuff. Are you saying that info is > too verbose? > > -Jay

Re: Recommended log level in prod environment.

We run at info too except when debugging stuff. Are you saying that info is too verbose? -Jay On Mon, Jul 22, 2013 at 6:43 PM, Calvin Lei wrote: > The beta release comes with mostly trace level logging. Is this > recommended? I notice our cluster produce way too many logs. I set all the > leve

Recommended log level in prod environment.

The beta release comes with mostly trace level logging. Is this recommended? I notice our cluster produce way too many logs. I set all the level to info currently.

Re: Messages TTL setting

Yes, all configuration changes should be possible to do one node at a time. -Jay On Mon, Jul 22, 2013 at 2:03 PM, arathi maddula wrote: > Hi, > > We have a 3 node Kafka cluster. We want to increase the maximum amount of > time for which messages are saved in Kafka data logs. > Can we change the

Messages TTL setting

Hi, We have a 3 node Kafka cluster. We want to increase the maximum amount of time for which messages are saved in Kafka data logs. Can we change the configuration on one node, stop it and start it and then change the configuration of the next node? Or should we stop all 3 nodes at a time, make co

Re: Logo

I actually did this the last time a logo was discussed :) https://docs.google.com/drawings/d/11WHfjkRGbSiZK6rRkedCrgmgFoP_vQ-QuWNENd4u7UY/edit As it turns out, it was a dung beetle in the book (I thought it was a roach as well). -David On 7/22/13 2:59

Re: Logo

It should be a roach in honor of Franz Kafka's Metamorphosis. On 7/22/2013 2:55 PM, S Ahmed wrote: Similar, yet different. I like it! On Mon, Jul 22, 2013 at 1:25 PM, Jay Kreps wrote: Yeah, good point. I hadn't seen that before. -Jay On Mo

Re: Logo

Similar, yet different. I like it! On Mon, Jul 22, 2013 at 1:25 PM, Jay Kreps wrote: > Yeah, good point. I hadn't seen that before. > > -Jay > > > On Mon, Jul 22, 2013 at 10:20 AM, Radek Gruchalski < > radek.gruchal...@portico.io> wrote: > > > 296 looks familiar: https://www.nodejitsu.com/ > >

Re: Replacing brokers in a cluster (0.8)

Here's a ruby cli that you can use to replace brokers...it shells out to the kafka-reassign-partitions.sh tool after figuring out broker lists from zk. Hope its useful. #!/usr/bin/env ruby require 'excon' require 'json' require 'zookeeper' def replace(arr, o, n) arr.map{|v| v == o ? n : v } e

Re: Replacing brokers in a cluster (0.8)

Is the kafka-reassign-partitions tool something I can experiment with now (this will only be staging data, in the first go-round). How does it work? Do I manually have to specify each replica I want to move? This would be cumbersome, as I have on the order of 100's of topicsOr does the tool

Re: Logo

296 looks familiar: https://www.nodejitsu.com/ Kind regards,
 Radek Gruchalski radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com) | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) | 
 ra...@gruchalski.com
 (mailto:ra...@gruchalski.com) 00447889948663



Re: Logo

Yeah, good point. I hadn't seen that before. -Jay On Mon, Jul 22, 2013 at 10:20 AM, Radek Gruchalski < radek.gruchal...@portico.io> wrote: > 296 looks familiar: https://www.nodejitsu.com/ > > Kind regards, > Radek Gruchalski > radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolo

Re: Replacing brokers in a cluster (0.8)

This seems like the type of behavior I'd ultimately want from the controlled shutdown tool . Currently, I believe the ShutdownBroker causes new leaders to be selected for any partition th

Logo

Hey guys, We need a logo! I got a few designs from a 99 designs contest that I would like to put forward: https://issues.apache.org/jira/browse/KAFKA-982 If anyone else would like to submit a design that would be great. Let's do a vote to choose one. -Jay

Re: Replacing brokers in a cluster (0.8)

A replica's data won't be automatically moved to another broker where there are failures. This is because we don't know if the failure is transient or permanent. The right tool to use is the kafka-reassign-partitions tool. It hasn't been thoroughly tested tough. We hope to harden it in the final 0.

Re: Apache Kafka Question

Millions of messages per day (with each message being few bytes) is not really 'Big Data'. Kafka has been tested for a million message per second. The answer to all your question IMO is "It depends". You can start with a single instance (Single machine installation). Let your producer send messag

Apache Kafka Question

Hi, I am planning to use Apache Kafka 0.8 to handle millions of messages per day. Now I need to form the environment, like (i) How many Topics to be created? (ii) How many partitions/replications to be created? (iii) How many Brokers to be created? (iv) How many consumer instances in consum

Replacing brokers in a cluster (0.8)

I'm planning to upgrade a 0.8 cluster from 2 old nodes, to 3 new ones (better hardware). I'm using a replication factor of 2. I'm thinking the plan should be to spin up the 3 new nodes, and operate as a 5 node cluster for a while. Then first remove 1 of the old nodes, and wait for the partitions