Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Hi Renato, Thanks for your feedback, they are very helpful! Please see my comments inline ... - About the consistency. 1. I guess when you are talking about consistency you mean versioning and thus making all tasks use a specific version (most of the time the last visible one), right? So there is

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Hi Wei, Thanks for the proposal, it looks good! I have some clarification questions though, hope you don't mind :) - About the consistency. 1. I guess when you are talking about consistency you mean versioning and thus making all tasks use a specific version (most of the time the last visible one

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks much for your comment, Jahadish! Please see inline for my response … -Wei On 5/19/17, 5:47 PM, "Jagadish Venkatraman" wrote: Thanks for writing up this proposal, Wei. This will go a long way in satisfying a number of Samza use-cases. I'm +1 to this idea. >> Section on

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks for writing up this proposal, Wei. This will go a long way in satisfying a number of Samza use-cases. I'm +1 to this idea. >> Section on proposed changes: Provide hooks to transform an incoming message to desired types (this is useful to store a subset of the incoming message). 1. I believ

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks for writing up this proposal, Wei. This will go a long way in satisfying a number of Samza use-cases. I'm +1 to this idea. >> Section on proposed changes: Provide hooks to transform an incoming message to desired types (this is useful to store a subset of the incoming message). 1. I believ

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks for writing up this proposal, Wei. This will go a long way in satisfying a number of Samza use-cases. I'm +1 to this idea. >> Section on proposed changes: Provide hooks to transform an incoming message to desired types (this is useful to store a subset of the incoming message). 1. I believ

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks Xinyu for your feedback. With regard to your question, when a new version of a file becomes available, we would already be in the normal processing mode, either the connector or external system would need to inject an indication to signal the end of the current version and continue send the

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Hi, Wei, +1 on the proposed design. This is going to reduce a lot of heavy-lifting work that's needed done by user code today to bootstrap a data stream into local store. The configs look quite straightforward and easy to set up. Overall the design looks great to me. I have one question: in the p

Re: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Thanks for trying 3 times, Wei. Sorry about the trouble. Not sure where the problem lies. Looking forward to review your design. Navina On Tue, May 16, 2017 at 8:56 AM, Wei Song wrote: > Hey everyone, > > I created a proposal for SAMZA-1278 > ,

Fwd: [DISCUSS] SEP-4: Adjunct Data Store for Unbounded DataSets

Hey everyone, I created a proposal for SAMZA-1278 , Adjunct Data Store for Unbounded DataSets, which introduces an automatic mechanism to store adjunct data for stream tasks. https://cwiki.apache.org/confluence/display/SAMZA/Adjunct+Da ta+Store+fo