Hi Sophie, Thanks very much for the response!
So if I understand correctly it will be impossible to avoid the repartition topic? e.g. my original message may have key = A...and will be partitioned on A. But in my Kafka Streams app, I will want to aggregate on A:B or A:C or A:D (B, C or D come from extra key values in the data)...but continue to partition on A. Then later read via REST all values for A. So to make this work I have to have a repartition topic even though I am not really repartitioning (i.e. all records for A should still be processed together). Is my understanding correct? So WindowedStreamPartitioner is a special case for avoiding the repartition topic? thanks in advance, Gareth On Wed, Mar 17, 2021 at 7:59 PM Sophie Blee-Goldman <sop...@confluent.io.invalid> wrote: > Hey Gareth, > > Kafka Streams state store partitioning is based on the partitioning of the > upstream input topics. > If you want your RocksDB stores to be partitioned based on the prefix of a > key, then you should > make sure the input topic feeding into it uses whatever partitioning > strategy you had in mind. > > If the source topics are user input topics and you have control over the > production to these topics, > then just use a custom partitioner to produce to them. If you don't have > control over them, you can > insert an intermediate/repartition topic between the input topics and the > subtopology with the RocksDB. > Check out the KStream#repartitioned operator, it accepts a Repartitioned > which itself accepts a > StreamPartitioner that you can use to control the partitioning. > > You can check out the class WindowedStreamPartitioner for an example: this > is how we handle the > WindowStore case that you pointed out. > > > > On Mon, Mar 15, 2021 at 11:45 AM Gareth Collins < > gareth.o.coll...@gmail.com> > wrote: > > > Hi, > > > > This may be a newbie question but is it possible to control the > > partitioning of a RocksDB KeyValueStore in Kafka Streams? > > > > For example, I perhaps only want to partition based on a prefix of a key > > rather than the full key. I assume something similar must be done for the > > WindowStore to partition without the window start time and sequence > number > > (otherwise window entries could be spread across partitions)? > > > > Sort of like the window store, I am wanting to be able to retrieve all > > values with a certain key prefix from the KeyValueStore with one read > > operation. > > > > Is this possible? > > > > thanks in advance, > > Gareth Collins > > >
