Re: Store changelog

2015-04-02 Thread Dan
Ah that makes sense now thanks Chris, re-reading that page it is clear. I think what confused me is this section from the configuration documentation for `stores.store-name.changelog`: "... Any output stream can be used as changelog, but you must ensure that only one job ever writes to a given cha

Re: Store changelog

2015-04-02 Thread Chinmay Soman
Also documented here: http://samza.apache.org/learn/documentation/0.9/container/state-management.html Check the "Local state in Samza" section - the diagram (and the description) explains this clearly. On Thu, Apr 2, 2015 at 10:36 AM, Chris Riccomini wrote: > Hey Dan, > > I think you might have

Re: Store changelog

2015-04-02 Thread Chris Riccomini
Hey Dan, I think you might have a misunderstanding in how changelogs work with Samza. Suppose you have a job with two tasks, and a single kv-store is configured with a changelog attached. The changelog, in Kafka, will have two partitions. Each task will use one partition of the changelog topic. Yo