Hey Gareth, Kafka Streams state store partitioning is based on the partitioning of the upstream input topics. If you want your RocksDB stores to be partitioned based on the prefix of a key, then you should make sure the input topic feeding into it uses whatever partitioning strategy you had in mind.
If the source topics are user input topics and you have control over the production to these topics, then just use a custom partitioner to produce to them. If you don't have control over them, you can insert an intermediate/repartition topic between the input topics and the subtopology with the RocksDB. Check out the KStream#repartitioned operator, it accepts a Repartitioned which itself accepts a StreamPartitioner that you can use to control the partitioning. You can check out the class WindowedStreamPartitioner for an example: this is how we handle the WindowStore case that you pointed out. On Mon, Mar 15, 2021 at 11:45 AM Gareth Collins <gareth.o.coll...@gmail.com> wrote: > Hi, > > This may be a newbie question but is it possible to control the > partitioning of a RocksDB KeyValueStore in Kafka Streams? > > For example, I perhaps only want to partition based on a prefix of a key > rather than the full key. I assume something similar must be done for the > WindowStore to partition without the window start time and sequence number > (otherwise window entries could be spread across partitions)? > > Sort of like the window store, I am wanting to be able to retrieve all > values with a certain key prefix from the KeyValueStore with one read > operation. > > Is this possible? > > thanks in advance, > Gareth Collins >
