Re: a few questions from high level consumer documentation.

On Thu, May 9, 2013 at 12:36 AM, Rob Withers wrote: > > > > -Original Message- > > From: Chris Curtin [mailto:curtin.ch...@gmail.com] > > > > 1 When you say the iterator may block, do you mean hasNext() may block? > > > > > > > Yes. > > Is this due to a potential non-blocking fetch (broke

Re: a few questions from high level consumer documentation.

On 5/9/13 8:27 AM, Chris Curtin wrote: On Thu, May 9, 2013 at 12:36 AM, Rob Withers wrote: -Original Message- From: Chris Curtin [mailto:curtin.ch...@gmail.com] 1 When you say the iterator may block, do you mean hasNext() may block? Yes. Is this due to a potential non-blocking f

Re: a few questions from high level consumer documentation.

Thanks, Neha On May 9, 2013 5:28 AM, "Chris Curtin" wrote: > > On Thu, May 9, 2013 at 12:36 AM, Rob Withers wrote: > > > > > > > > -Original Message- > > > From: Chris Curtin [mailto:curtin.ch...@gmail.com] > > > > > > 1 When you say the iterator may block, do you mean hasNext() may block

Re: can the replication factor for a topic be changed after it's created?

I think so but this hasnt been tested. Thanks, Neha On May 8, 2013 11:52 PM, "Jason Rosenberg" wrote: > so does that, thereby, allow you to increase the replication for a > partition? > > > On Wed, May 8, 2013 at 10:48 PM, Neha Narkhede >wrote: > > > If you add more brokers to an existing clust

key used by producer

Hi, I am looking at the example producer code for kafka 0.8. I notice it is possible to specify a key when creating KeyedMessage. This key will be used for assigning the message to some partition. I wonder if the key will be received by the consumer. Thanks. Libo

Re: [0.8] exception running on windows when attempting to send to an uncreated topic

You are missing the jopt-simple jar. Thanks, Jun On Wed, May 8, 2013 at 11:03 PM, Rob Withers wrote: > I am running on windows. I am programmatically (no scripts) starting a zk, > 2 brokers, 2 consumers and a producer, in this order but the first 3 at > once, then the other 3 at once, all wi

Re: Can't connect to a server if not enough partitions

Currently, when creating a topic, we require the number of live brokers to be greater than the replication factor. Once the topic is created, the number of live brokers can be less than the replication factor. Thanks, Jun On Wed, May 8, 2013 at 11:50 PM, Jason Rosenberg wrote: > Neha, > > Tha

Re: can the replication factor for a topic be changed after it's created?

Currently, we don't allow the replication factor of an existing topic/partition to be changed. We plan to add a tool to do that in the future. Thanks, Jun On Wed, May 8, 2013 at 11:52 PM, Jason Rosenberg wrote: > so does that, thereby, allow you to increase the replication for a > partition?

Re: key used by producer

Yes, our consumer iterator returns MessageAndMetadata. You can do consumerIterator.next().key() to get the key. Thanks, Jun On Thu, May 9, 2013 at 7:44 AM, Yu, Libo wrote: > Hi, > > I am looking at the example producer code for kafka 0.8. > I notice it is possible to specify a key when creati

Re: key used by producer

Yes, I'm pretty sure keys are now retained with the message and returned in the consumer. In the Java/Scala client, ConsumerIterator returns a MessageAndMetadata which includes key, message, topic, partition, and offset -David On 5/9/13 10:44 AM, Yu, Libo wrote: Hi, I am looking at the exa

exception report

Hi, When I tried to launch the server, I got this exception: [2013-05-09 11:38:04,106] INFO Loading log 'lost+found' (kafka.log.LogManage r) [2013-05-09 11:38:04,108] FATAL Fatal error during KafkaServerStable startup . Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.StringIndexO

Problem with consumers in a consumers group

Hi. I`m sending nginx logs to kafka. Each nginx goes to one exclusive partition. I guarantee this by asigning to each nginx a number (1 to 22), and I use that number for a custom partitoner's partition method as the key parameter: return Integer.parseInt(key) % partitions I have 22 nginxs, so I

Re: exception report

The topic name should only consist of ascii characters and '_' and '-'. I agree that this error message is misleading though. Which version of Kafka are you running ? Thanks, Neha On Thu, May 9, 2013 at 8:44 AM, Yu, Libo wrote: > Hi, > > When I tried to launch the server, I got this exception:

Re: can the replication factor for a topic be changed after it's created?

Jun, the question is whether or not you can increase the replication factor of a partition through the partition reassignment tool or not. I think you can but I'm not sure until I test this. Thanks, Neha On Thu, May 9, 2013 at 7:58 AM, Jun Rao wrote: > Currently, we don't allow the replication

Re: can the replication factor for a topic be changed after it's created?

Yes, your are right. Thanks, Jun On Thu, May 9, 2013 at 9:07 AM, Neha Narkhede wrote: > Jun, the question is whether or not you can increase the replication factor > of a partition through the partition reassignment tool or not. I think you > can but I'm not sure until I test this. > > Thanks,

Re: Problem with consumers in a consumers group

Could you check if the topic name is the same on all 5 brokers ? Can you send around the contents of the kafka log directory on all 5 brokers ? Thanks, Neha On Thu, May 9, 2013 at 8:51 AM, Javier Iseruk < javier.ise...@mercadolibre.com> wrote: > Hi. > I`m sending nginx logs to kafka. Each nginx

Re: exception report

Hmm, that looks a lot like the lost+found directory on a Linux file system. Is your Kafka data directory a mount point that has a lost and found directory? On Thu, May 9, 2013 at 12:10 PM, Neha Narkhede wrote: > The topic name should only consist of ascii characters and '_' and '-'. I > agree th

Re: Problem with consumers in a consumers group

Thanks for the answer. I send you the info: I am ommiting all the files in each directory, just leaves one for showing the structure. There are two topics: access_log@nginx-nginx-api-master and access_log@nginx-nginx-api-v1. The consumer uses a whitelist with the regex access_log@nginx-nginx-api-.*

Re: Problem with consumers in a consumers group

Hi again! I have more info to share. What I'm seeing in zookeeper is that the consumers are owning almost the same partitions quantity, no matter if that partition has data or not. So for example: Consumer 1: 17 partitions (all with data) Consumer 2: 17 partitions (only five with data) Consumer 3:

bug report

Hi, I created a volume and mounted it to /data/kafka. There is a lost+found/ dir in it. This crashed the server. [2013-05-09 16:58:53,607] INFO Loading log 'lost+found' (kafka.log.LogManager) [2013-05-09 16:58:53,609] FATAL Fatal error during KafkaServerStable startup. Pre pare to shutdown (kafka