This may be a bit use-case dependent, but I think simply using the key from the Kafka record as the KV key is a good start. Another option is to use topic-partition-offset as the key. This has the benefit of removing duplicates, but it also means that keys are no longer meaningful for applications querying the KV store.
The nice thing is that now that we are adding transformations of Connect, you can build one solution and users who don't like it can use transformations to modify the key. Gwen On Tue, Jan 10, 2017 at 8:02 PM, Nilkanth Patel <nilkanth.hpa...@gmail.com> wrote: > Hello, > > I am writing a kafka sink connector for my product that is distributed > table (Underneath a distribute K-V store where cluster of nodes are holding > a different partitions/buckets of a table. This is based on Hash > partitioned on Keys). > > When i write a SinkTask, I get the SinkRecord that contains both key and > value. > > Can some one suggest/outline the general guidelines for keys to be used > with K-V store from the SinkRecord. > > What should be the key for external K-V store to be used to store a records > from kafka topics to external K-V store. > > Thanks, > Nilkanth. -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
