looks like you need to use keyedPartitioner option(partitionerType =3). default is cyclic partitioning.
https://www.npmjs.com/package/kafka-node#highlevelproducer On Mon, Oct 23, 2017 at 9:24 PM, <eazama...@gmail.com> wrote: > Please correct me if I've misunderstood something, but your keys being > Subject0, Subject1, Subject2, etc, mean you don't actually have identical > keys, right? If that's the case then you wouldn't expect to see all of the > messages in a single partition. > > > On Oct 23, 2017, at 8:43 AM, Andrea Giordano < > andrea.giordano....@gmail.com> wrote: > > > > The output of: > > ./kafka_2.11-0.11.0.0/bin/kafka-topics.sh --zookeeper localhost:2181 > --list > > > > LCacc > > LCgyr > > LTacc > > LTgyr > > RCacc > > RCgyr > > RTacc > > RTgyr > > __consumer_offsets > > > > output sample for a topic (all equals) > > ./kafka_2.11-0.11.0.0/bin/kafka-topics.sh --describe --zookeeper > localhost:2181 --topic LCacc > > > > Topic:LCacc PartitionCount:10 ReplicationFactor:1 Configs: > > Topic: LCacc Partition: 0 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 1 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 2 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 3 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 4 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 5 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 6 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 7 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 8 Leader: 0 Replicas: 0 Isr: 0 > > Topic: LCacc Partition: 9 Leader: 0 Replicas: 0 Isr: 0 > > > > the kafka producer is implemented in node-red, so the code about it is > in javascript. > > look at the link for it: https://pastebin.com/Rqd2Q3kx > > > > I use Flink java to get kafka messages and implementing a deserialiser > effectively I see my key string in key kafka message field (so I’m “quite” > sure the javascript code is correctly implemented). > > > > Finally my keys are something like: > > > > Subject0 > > Subject1 > > Subject2 > > … and so on. > > > > > > Thank you for the help, > > Andrea > > > > > > > > > >> On 23 Oct 2017, at 17:25, Manikumar <manikumar.re...@gmail.com> wrote: > >> > >> Can you post the sample code? > >> > >> On Mon, Oct 23, 2017 at 8:53 PM, Andrea Giordano < > >> andrea.giordano....@gmail.com> wrote: > >> > >>> Hi, > >>> I set a Kafka broker with some topics and where each topic is divided > into > >>> 10 partitions. > >>> As I understood on Kafka doc, if I send a keyed message to kafka the > key > >>> is used to distribute the message in a particular partition (using an > hash > >>> value of the key): in this way messages with same key go in the same > >>> partition. > >>> > >>> I’m trying to run the broker sending messages with the same key and I > >>> noted that executing the following command: > >>> > >>> bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list > >>> localhost:9092 --topic LCacc --time -1 > >>> > >>> > >>> I obtained the output: > >>> > >>> Honestly I expected to see just a partition full (due to the only one > key > >>> used). Instead it seems kafka distributed message in a quite uniform > way. > >>> Is this normal? > >>> I tried also to send messages with different keys (for example sending > >>> much more messages with a key respect to another) but nothing changed. > >>> > >>> LCacc:8:60699 > >>> LCacc:2:60737 > >>> LCacc:5:60716 > >>> LCacc:4:60722 > >>> LCacc:7:60701 > >>> LCacc:1:60744 > >>> LCacc:9:60695 > >>> LCacc:3:60731 > >>> LCacc:6:60710 > >>> LCacc:0:61967 > >>> > >>> > > >
