Hey Dan, I think you might have a misunderstanding in how changelogs work with Samza. Suppose you have a job with two tasks, and a single kv-store is configured with a changelog attached. The changelog, in Kafka, will have two partitions. Each task will use one partition of the changelog topic. You only need one topic per-changelog (and no prefix) because there are multiple partitions per changelog, and there's a 1:1 mapping between a task and its changelog partition.
Cheers, Chris On Thu, Apr 2, 2015 at 10:30 AM, Dan <danharve...@gmail.com> wrote: > Hi all, > > We're just starting out using Samza to process streams we've already got in > Kafka. Some of the jobs we've written are using the per task KV store which > are being persisted to a changelog topic in Kafka. As you need a different > changelog topic per task we are wondering how people are dealing with > ensuring that each task's store has a separate changelog. > > I think we could define multiple stores in the properties file, then pick > the correct one for each task index. But that seems quite a verbose way to > go about that? > > If Samza could use a prefix in the properties file then generate a topic > name for each task it would simplify using that. Maybe there's something > I'm missing from this? > > Thanks, > Dan >
