Re: Partition id of consumed message?

Thank you. but I dont see that method. I dont know whats wrong On Wed, Nov 26, 2014 at 6:31 PM, Jun Rao wrote: > Yes, the consumer iterator returns MessageAndMetadata which contains a > method partition(). > > Thanks, > > Jun > > On Wed, Nov 26, 2014 at 10:40 AM, Palur Sandeep > wrote: > > > Hi

Re: Partition key not working properly

The issue is probably the hashcode of byte[] is based on reference, instead of the value. You can try binding the producer to [Array[byte],Array[byte]] and using ByteArrayPartitioner. Alternatively, you can bind the producer to [String,Array[byte]] and the default partitioner should work. Thanks,

Re: Partition id of consumed message?

Yes, the consumer iterator returns MessageAndMetadata which contains a method partition(). Thanks, Jun On Wed, Nov 26, 2014 at 10:40 AM, Palur Sandeep wrote: > Hi all, > > I am using kafka version 0.8.1.1. In the consumer side, can I find out the > partition id of each message I am consuming u

Re: How many messages does each broker have?

Hi Jiangle, Thanks for the information. This is what I get when I print MessageandMetadata *Thread 0: Message(magic = 0, attributes = 0, crc = 127991357, key = java.nio.HeapByteBuffer[pos=0 lim=1 cap=55], payload = java.nio.HeapByteBuffer[pos=0 lim=50 cap=50])* Can you please tell me where can

RE: Partition key not working properly

Hi François, I agree with you and Svante, so I think my logic is correct. But I really can't find why my problem happens and I was stuck here for weeks. I think posting some of my codes might be helpful, could you please give the codes a quick check? Here is the producer codes, I didn't use th

Kafka 0.8.2 release - before Santa Claus?

Hi, People using SPM to monitor Kafka have been anxiously asking us about the 0.8.2 release and we've been telling them December. Is that still the plan? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Partition key not working properly

Hi haoming, As far as I know, svante is right. Maybe you modified your default partitioner? or are you sure the same key go to different partitions? maybe its just 2 keys that are going to the same partition? Because it's possible that you have something like that - key "1" -> partition 3 - key

Re: MetadataResponse error code handling

Thanks, that's exactly what I needed to know. Evan On Wed, Nov 26, 2014 at 2:22 PM, Guozhang Wang wrote: > For the new producer, the only error code that will result in invalid > partition metadata is LeaderNotAvailable, and as for the implementation > details of the KAFKA-1609 patch, it handle

Re: How many messages does each broker have?

Hi Sandeep, For old producer, I don’t think you can achieve strict even distribution of messages across partitions within the same topic. But You can potentially reduce the sticking time by setting topic.metadata.refresh.interval.ms to be lower, e.g. 1 second. Kafka-544 added the partition inform

Re: MetadataResponse error code handling

For the new producer, the only error code that will result in invalid partition metadata is LeaderNotAvailable, and as for the implementation details of the KAFKA-1609 patch, it handles this error code by setting the leader field to null. Guozhang On Wed, Nov 26, 2014 at 11:08 AM, Evan Huus wrot

Re: MetadataResponse error code handling

Hi Guozhang, thanks for the information. I am implementing/maintaining the Golang producer [1] so I do need to worry about these details. You mention LeaderNotAvailable as the only one, but the JIRA ticket I linked also talks about ReplicaNotAvailable. Are both of these ignored? Or is it that Lea

Partition id of consumed message?

Hi all, I am using kafka version 0.8.1.1. In the consumer side, can I find out the partition id of each message I am consuming using high level consumer API? Thanks, Sandeep -- Regards, Sandeep Palur Data-Intensive Distributed Systems Laboratory, CS/IIT Department of Computer Science, Illinois

Re: Can Mirroring Preserve Every Topic's Partition?

Hello Alex, This can be done by doing some tweaks in the MM code (with the 0.8.2 new producer). 1. Set-up your MM to have the total # of producers equal to the #. of partitions in source / target cluster. 2. When the consumer of the MM gets a message, put the message to the producer's queue base

Re: How many messages does each broker have?

Hi Jiangjie, I am using the high level consumer (ZookeeperConsumerConnector), after getting the message from stream, but I don't see this "message.Partition". Please help me how to get the partition id form message. What is that I can to do get messages evenly distributed among partitions? do you

Re: MetadataResponse error code handling

Hi Evan, The error code is per-partition, and for new producer only LeaderNotAvailable error will cause it to ignore the metadata for that partition. If you are using the producer clients directly you would not need to worry about these error handling details; if you are fetching the metadata usin

Re: How many messages does each broker have?

Hi Sandeep, If you are sending messages to different topics, each topic will stick to a random partition for 10 min. Since they are likely sticking to different brokers, you will still see messages roughly evenly distributed. If you are using high level consumer (ZookeeperConsumerConnector), after

Re: [DISCUSSION] adding the serializer api back to the new java producer

Yes, that will be a separate release. Possibly an 0.8.2 beta-2, followed by 0.8.2 final. Thanks, Jun On Wed, Nov 26, 2014 at 2:24 AM, Shlomi Hazan wrote: > Jay, Jun, > Thank you both for explaining. I understand this is important enough such > that it must be done, and if so, the sooner the be

Re: Kafka High Level Consumer Connector shuts down after 10 seconds

Hi, you are programatically shutting down the executor after 10 seconds try { Thread.sleep(1); } catch (InterruptedException ie) { } example.shutdown(); if you do not execute this code your threads will run forever. Davide B. --

Re: [DISCUSSION] adding the serializer api back to the new java producer

Jay, Jun, Thank you both for explaining. I understand this is important enough such that it must be done, and if so, the sooner the better. How will the change be released? a beta-2 or release candidate? I think that if possible, it should not overrun the already released version. Thank you guys fo