Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

> Seems like they're (usually) going to be used by the framework, are pretty simple to write, and could probably be written as a common Util method if we find them repetitive. Hmm.. To be honest, I didn't see the value of it too. When Xinyu suggested this change, I assumed it was the newly accept

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi Navina, 1. Assuming the environment can put the processor ID in the config, ProcessorIdGenerator#generateProcessorId(Config config) makes sense. Passing all of Config is rather broad, but I don't think we have an environment specific subset class for config yet, so should be OK. 2. I don't yet

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi Prateek, > 1. Do you have any examples of custom processor IDs? Wondering what information/classes ProcessorIdGenerator would need to be able to generate one. Yeah. When I was trying to implement the proposal, I was wondering the same thing as well. However, it might end up being specific to the

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi Navina, Thanks for SEP-1, looks pretty good to me. A few questions/comments: Implementation/Interface related: 1. Do you have any examples of custom processor IDs? Wondering what information/classes ProcessorIdGenerator would need to be able to generate one. 2. The default "static" getProcesso

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, I have updated the SEP based on all the feedback. Feel free to comment. I will start the [vote] mail thread, if there are no further questions within the next 24 hours. Thanks! Navina On T

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, I have updated the SEP based on all the feedback. Feel free to comment. I will start the [vote] mail thread, if there are no further questions within the next 24 hours. Thanks! Navina On T

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi Jagadish, Thanks for the suggestion. You are right in that it should be the responsibility of the JobCoordinator to assign identifiers. > 'm only wondering if this logic could instead reside inside the Job Coordinator (which is internal to the StreamProcessor) instead of relying on something ex

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Thanks for writing this SEP! Here's an alternate approach instead of taking the "String processorId" as a parameter in the constructor. In my view, the "processorId" could be generated by the StreamProcessor internally (instead of being generated up-stream and passed in). The Job Coordinator API c

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

> I am working on the ApplicationRunner SEP right now. Will send out the discussion email once I am done. Perfect! :) On Thu, Mar 16, 2017 at 5:13 PM, xinyu liu wrote: > Right, the static factory is very simple as you said. It's pretty > convenient for the client to use. > > I am working on the

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Right, the static factory is very simple as you said. It's pretty convenient for the client to use. I am working on the ApplicationRunner SEP right now. Will send out the discussion email once I am done. Thanks, Xinyu On Thu, Mar 16, 2017 at 4:50 PM, Navina Ramesh (Apache) wrote: > > One minor

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

> One minor thing I found is that the name of the config is camel case (*processor.idGenerator.class*). Seems Samza's practice is to use all lower case configs with "." delimiter. Do you think we should stick to this convention? I am always torn between the "convention" we have and the better way

Re: [DISCUSS] SEP-1: Semantics of ProcessorId in Samza

The proposal looks great to me! Changing the id type to string will make sure this can work with other types of cluster which doesn't support integer id. The interface and config provides a pluggable way to have different id generators for different use cases. One minor thing I found is that the na

[DISCUSS] SEP-1: Semantics of ProcessorId in Samza

Hi everyone, I created a proposal for SAMZA-1126, which addresses the semantics of ProcessorId in Samza. For most purposes, ProcessorId is same as the logical id that Samza assigns for each Yarn container. It is primarily used in JobModel as a key for the corresponding ContainerModel and also, in c