Re: Topicmetadata response miss some partitions information sometimes

Yeah, but that gives them all the partitions and does not differentiate between available vs unavailable right. Thanks, Mayuresh On Thu, Mar 5, 2015 at 9:14 AM, Guozhang Wang wrote: > I think today people can get the available partitions by calling > partitionsFor() API, and iterate the partit

Re: Topicmetadata response miss some partitions information sometimes

I think today people can get the available partitions by calling partitionsFor() API, and iterate the partitions and filter those whose leader is null, right? On Wed, Mar 4, 2015 at 4:40 PM, Mayuresh Gharat wrote: > Cool. So then this is a non issue then. To make things better we can expose > th

Re: Topicmetadata response miss some partitions information sometimes

Cool. So then this is a non issue then. To make things better we can expose the availablePartitons() api through Kafka producer. What do you think? Thanks, Mayuresh On Tue, Mar 3, 2015 at 4:56 PM, Guozhang Wang wrote: > Hey Jun, > > You are right. Previously I thought only in your recent patch

Re: Topicmetadata response miss some partitions information sometimes

Hey Jun, You are right. Previously I thought only in your recent patches you add the partitionWithAvailableLeaders that this gets exposed, however it is the opposite case. Guozhang On Tue, Mar 3, 2015 at 4:40 PM, Jun Rao wrote: > Guozhang, > > Actually, we always return all partitions in the m

Re: Topicmetadata response miss some partitions information sometimes

Guozhang, Actually, we always return all partitions in the metadata response whether the leaders are available or not. Thanks, Jun On Sat, Feb 28, 2015 at 10:46 PM, Guozhang Wang wrote: > Hi Honghai, > > 1. If a partition has no leader (i.e. all of its replicas are down) it will > become offl

Re: Topicmetadata response miss some partitions information sometimes

Filed as https://issues.apache.org/jira/browse/KAFKA-1998 Evan On Mon, Mar 2, 2015 at 5:19 PM, Guozhang Wang wrote: > That is a valid point, today the returned metadata response already > contains partitions even with error code, so we can expose that in the > Cluster / KafkaProducer class. Cou

Re: Topicmetadata response miss some partitions information sometimes

That is a valid point, today the returned metadata response already contains partitions even with error code, so we can expose that in the Cluster / KafkaProducer class. Could you file a JIRA? Guozhang On Sun, Mar 1, 2015 at 7:11 PM, Evan Huus wrote: > Which I think is my point - based on my cu

Re: Topicmetadata response miss some partitions information sometimes

Which I think is my point - based on my current understanding, there is *no* way to find out the total number of partitions for a topic besides hard-coding it or manually reading it from zookeeper. The kafka metadata API does not reliably expose that information. Evan On Sun, Mar 1, 2015 at 10:07

Re: Topicmetadata response miss some partitions information sometimes

I see. If you need to make sure messages are going to the same partition during broker bouncing / failures, then you should not depend on the partitioner to decide the partition id but explicitly set it before calling send(). For example, you can use the total number of partitions for the topic, n

Re: Topicmetadata response miss some partitions information sometimes

My concern is more with the partitioner that determines the partition of the message. IIRC, it does something like "hash(key) mod #partitions" in the normal case, which means if the # of partitions changes because some of them are offline, then certain messages will be sent to the wrong (online) pa

Re: Topicmetadata response miss some partitions information sometimes

Evan, In the java producer, partition id of the message is determined in the send() call and then the data is appended to the corresponding batch buffer (one buffer for each partition), i.e. the partition id will never change once it is decided. If the partition becomes offline after this, the sen

Re: Topicmetadata response miss some partitions information sometimes

On Sun, Mar 1, 2015 at 1:46 AM, Guozhang Wang wrote: > Hi Honghai, > > 1. If a partition has no leader (i.e. all of its replicas are down) it will > become offline, and hence the metadata response will not have this > partition's info. > If I am understanding this correctly, then this is a probl

Re: Topicmetadata response miss some partitions information sometimes

Hi Honghai, 1. If a partition has no leader (i.e. all of its replicas are down) it will become offline, and hence the metadata response will not have this partition's info. 2. Any of the brokers cache metadata and hence can handle the metadata request. It's just that their cache are updated async

Topicmetadata response miss some partitions information sometimes

We have one topic with 4 partitions, but sometimes only get metadata of 2 partitions, did anyone meet this kind of situation before?If some partition has no leader at that moment, will it cause this problem? How to make some partition has no leader?If 6 brokers has some partitions of the top