Hi, Thank you for answer.
How is the balancing done without zookeeper? Is it done locally to producer thread with round robin, or do we need to use a VIP with a TCP balancer for each partition, with the hosts having that partition replicas, or by other means? Indeed, the producers reconnect, and increasing the maximum number of send message attempts we achieved the wanted effect, namely to queue messages even when kafka brokers are down and, the latest on producer.close(), wait for them to come back up and send them the messages. We close it because we have a map reduce for that backend process, not a streaming one, because storm in-memory queuing limitations. When Samza comes up with persistent channels based on kafka this will likely become a streaming process with no close involved usually. We use an async producer. By blocking I meant the write in its queue from the client is blocking when the queue is full, as we don't want to loose them (enque.ms < 0). Thank you again, Nicu Marasoiu Adobe On 9/27/13 5:57 PM, "Jun Rao" <jun...@gmail.com> wrote: >Yes, the zk option is no longer available in the producer in 0.8 to >simplify the logic in the client. Load balancing in the producer is still >supported though. > >I am not sure what you mean by "blocking async" producer. The producer has >an async mode, but then the send() calls are not blocking. The expected >behavior is that if all brokers are down, all sends will fail. However, >when the brokers are up again, sends will succeed again. There is no need >to close the producer in between. Is that not what you are seeing? > >Thanks, > >Jun > > >On Fri, Sep 27, 2013 at 7:15 AM, Nicolae Marasoiu ><nmara...@adobe.com>wrote: > >> Hi, >> >> We have a blocking async producer. >> We noticed: >> >> 1. when the metadata brokers are down, the client no longer attempts >>to >> reconnect to any of them. >> 2. zookeeper.connect is no longer recognized as a valid configuration, >> hinting that in fact the zookeeper balancing is no longer supported on >> producing? >> >> The net effect is that producer.close() immediately returns while kafka >> cluster is down. Our expectation is to remain waiting until some nodes >>go >> up and try to send the data. >> >> I know that in 0.7.x this was the behaviour, based on zookeeper >> whitelisting, balancing and watching. >> >> Please advice, >> Nicu Marasoiu, Adobe >>
