Yes I understand that.
The streams application takes care of that
when I do:
input
.map(new KeyValueMapper<K, V, KeyValue<K, V1>>() {
public KeyValue<K, V1> apply(K key, V value) {
......
return new KeyValue<K, V1>(new_key, new_value);
}
}).through(k_serde, v_serde, "destination-topic");
It ensures that for a particular new_key it will send that new_value to a
specific partition of destination-topic.
My question is that when we increase number of partitions for
destination-topic, how can we ensure that all old values for a particular
new_key and new values for same new_key are logged to same partition.
Because if streams decide to change the partition for some old now (once we
have increased the partitions), how can we ensure old data is also present
in the same partition.
Is this handled automatically or do we have to do some kind of topic
migration (or if this is something we cannot guarantee to do?)
Thanks
Sachin
On Mon, Aug 21, 2017 at 12:15 PM, Zafar Ansari <zafar142...@gmail.com>
wrote:
> Hi
> You can specify a partition function while producing a message to Kafka
> brokers. This function will determine which partition the message should be
> sent to.
> See
> https://edgent.apache.org/javadoc/r1.1.0/org/apache/
> edgent/connectors/kafka/KafkaProducer.html#publish-
> org.apache.edgent.topology.TStream-org.apache.edgent.
> function.Function-org.apache.edgent.function.Function-org.
> apache.edgent.function.Function-org.apache.edgent.function.Function-
>
>
>
> On 21 August 2017 at 12:02, Sachin Mittal <sjmit...@gmail.com> wrote:
>
> > Hi,
> > I have a topic which has four partitions and data is distributed among
> > those based on a specified key.
> >
> > If I want to increase the number of partitions to six how can I do the
> same
> > and also making sure that messages for a given key always go to one
> > (specific) partition only.
> >
> > Will the existing message redistribute themselves among new partition.
> >
> > Also say earlier message of key A went to partition 1 and going forward
> any
> > new message go to same partition where earlier messages for that key are?
> >
> > And by increasing partitions some keys may use a different partition now,
> > so how do I ensure the case of all messages of that key belong to single
> > partition.
> >
> > Thanks
> > Sachin
> >
>
