What is your partitioning function?
François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure <http://www.etsmtl.ca/> Capitaine Club Capra <http://capra.etsmtl.ca/> VP-Communication - CS Games <http://csgames.org> 2014 Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> Comité Organisateur Olympiades ÉTS 2012 Compétition Québécoise d'Ingénierie 2012 - Compétition Senior On Mon, Aug 25, 2014 at 6:18 AM, Ashutosh Prasad Panigrahi < ashutosh1...@gmail.com> wrote: > Hi Guozhang, > > I repeated my experiment with topic.metadata.refresh.interval.ms set to 60 > seconds, and executed the mirror for more than 24 hours. > > I still see that it only writes to Partition "1" and totally ignores > partition "0". > > Can you please provide me some pointers to debug this. > > Ashutosh > > > On Thu, Aug 21, 2014 at 8:52 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hi Ashutosh, > > > > Maybe this faq can help you? > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified > > ? > > > > Guozhang > > > > > > On Thu, Aug 21, 2014 at 12:09 AM, Ashutosh Prasad Panigrahi < > > ashutosh1...@gmail.com> wrote: > > > > > Hello All, > > > > > > When i start the mirroring process, events are written only to > partition > > > "1" on the target cluster and no data is written to partition 0. > > > > > > I am stuck on this and wondering what am i doing wrong . > > > > > > My production setup looks something like this > > > 1. Datacenter1 : 3 kafka nodes/3 zookeeper nodes > > > 2. Datacenter 2 : 3 kafka nodes/3 zookeeper nodes > > > > > > All the kafka nodes are configured with auto.create.topics.enable=true > > and > > > the default partition is 2 . > > > > > > I start the kafka mirroring tool with the following command > > > > > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config > > > mirror-maker/consumer.properties --producer.config > > > mirror-maker/producer.properties --whitelist=".*" *--num.streams 2 > > > --num.producers 2* > > > > > > > > > the consumer.properties for mirror-config looks like this > > > group.id=test-group > > > zookeeper.connect=<ZK_IP_LIST> > > > > > > and producer.properties for mirror-config looks like this > > > > > > metadata.broker.list=<KAFKA_IP_LIST> > > > producer.type=async > > > topic.metadata.refresh.interval.ms=-1 > > > > > > > > > > > > Regards > > > -- > > > Ashutosh > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > Ashutosh Prasad Panigrahi > > All you who sleep tonight far from the ones you love > No hands to the left or right and emptiness above > Know that you aren't alone > the whole world shares your tears > some for two nights or one > and some for all their years....... >
